[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384993574762-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 01:09:54 dist-test-slave-7b22 krb5kdc[399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 01:09:54 dist-test-slave-7b22 krb5kdc[399](info): set up 2 sockets
Sep 26 01:09:54 dist-test-slave-7b22 krb5kdc[399](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260926 01:09:55.462082 392 mini_kdc.cc:121] Time spent starting KDC: real 1.865s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384993574762-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384993574762-392-0/krb5kdc/test-user.keytab.
Sep 26 01:09:55 dist-test-slave-7b22 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384995, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T01:09:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:09:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384993574762-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384993574762-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 01:09:55.585747 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384993574762-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384993574762-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384993574762-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45905
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384993574762-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45905
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:09:55.666702 412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:09:55.666935 412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:09:55.666966 412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:09:55.668197 412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:09:55.668244 412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:09:55.668262 412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:09:55.668278 412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:09:55.668292 412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:09:55.669791 412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384993574762-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384993574762-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45905
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384993574762-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45905
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384993574762-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:09:55.670079 412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:09:55.670316 412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:09:55.675333 417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:09:55.675339 420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:09:55.675567 418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:09:55.675755 412 server_base.cc:1034] running on GCE node
I20260926 01:09:55.675994 412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:09:55.676290 412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:09:55.677441 412 hybrid_clock.cc:648] HybridClock initialized: now 1790384995677446 us; error 88 us; skew 500 ppm
Sep 26 01:09:55 dist-test-slave-7b22 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384995, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:09:55.680083 412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 01:09:55.680624 412 webserver.cc:460] Webserver started at http://127.0.98.62:33441/ using document root <none> and password file <none>
I20260926 01:09:55.680827 412 fs_manager.cc:359] Metadata directory not provided
I20260926 01:09:55.680874 412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:09:55.681001 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:09:55.682242 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384993574762-392-0/minicluster-data/master-0/data/instance:
uuid: "7243178f18ce464b8b195ee0364ee5af"
format_stamp: "Formatted at 2026-09-26 01:09:55 on dist-test-slave-7b22"
server_key: "14225b48a9c9c8419468596c1801414b"
server_key_iv: "0625906202b49b315784cb7d80bfafb8"
server_key_version: "encryptionkey@0"
I20260926 01:09:55.682586 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384993574762-392-0/minicluster-data/master-0/wal/instance:
uuid: "7243178f18ce464b8b195ee0364ee5af"
format_stamp: "Formatted at 2026-09-26 01:09:55 on dist-test-slave-7b22"
server_key: "14225b48a9c9c8419468596c1801414b"
server_key_iv: "0625906202b49b315784cb7d80bfafb8"
server_key_version: "encryptionkey@0"
I20260926 01:09:55.683928 412 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:09:55.684752 427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:09:55.684952 412 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:09:55.685041 412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384993574762-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384993574762-392-0/minicluster-data/master-0/wal
uuid: "7243178f18ce464b8b195ee0364ee5af"
format_stamp: "Formatted at 2026-09-26 01:09:55 on dist-test-slave-7b22"
server_key: "14225b48a9c9c8419468596c1801414b"
server_key_iv: "0625906202b49b315784cb7d80bfafb8"
server_key_version: "encryptionkey@0"
I20260926 01:09:55.685103 412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384993574762-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384993574762-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384993574762-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:09:55.701524 412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:09:55.703820 412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:09:55.703981 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:09:55.707841 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45905
I20260926 01:09:55.707878 479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45905 every 8 connection(s)
I20260926 01:09:55.708221 412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
I20260926 01:09:55.709121 480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:09:55.710860 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 412
I20260926 01:09:55.710987 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384993574762-392-0/minicluster-data/master-0/wal/instance
I20260926 01:09:55.711763 392 external_mini_cluster.cc:1428] Setting key 3e08716283e3e26bbe427346322b6b61
I20260926 01:09:55.711978 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7243178f18ce464b8b195ee0364ee5af: Bootstrap starting.
I20260926 01:09:55.712680 480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7243178f18ce464b8b195ee0364ee5af: Neither blocks nor log segments found. Creating new log.
I20260926 01:09:55.712988 480 log.cc:824] T 00000000000000000000000000000000 P 7243178f18ce464b8b195ee0364ee5af: Log is configured to *not* fsync() on all Append() calls
I20260926 01:09:55.713703 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7243178f18ce464b8b195ee0364ee5af: No bootstrap required, opened a new log
I20260926 01:09:55.715461 480 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7243178f18ce464b8b195ee0364ee5af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7243178f18ce464b8b195ee0364ee5af" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45905 } }
I20260926 01:09:55.715601 480 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7243178f18ce464b8b195ee0364ee5af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:09:55.715632 480 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7243178f18ce464b8b195ee0364ee5af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7243178f18ce464b8b195ee0364ee5af, State: Initialized, Role: FOLLOWER
I20260926 01:09:55.715778 480 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7243178f18ce464b8b195ee0364ee5af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7243178f18ce464b8b195ee0364ee5af" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45905 } }
I20260926 01:09:55.715847 480 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7243178f18ce464b8b195ee0364ee5af [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:09:55.715878 480 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7243178f18ce464b8b195ee0364ee5af [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:09:55.715924 480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7243178f18ce464b8b195ee0364ee5af [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:09:55.716743 480 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7243178f18ce464b8b195ee0364ee5af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7243178f18ce464b8b195ee0364ee5af" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45905 } }
I20260926 01:09:55.716838 480 leader_election.cc:302] T 00000000000000000000000000000000 P 7243178f18ce464b8b195ee0364ee5af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7243178f18ce464b8b195ee0364ee5af; no voters:
I20260926 01:09:55.716993 480 leader_election.cc:288] T 00000000000000000000000000000000 P 7243178f18ce464b8b195ee0364ee5af [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:09:55.717027 485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7243178f18ce464b8b195ee0364ee5af [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:09:55.717119 485 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7243178f18ce464b8b195ee0364ee5af [term 1 LEADER]: Becoming Leader. State: Replica: 7243178f18ce464b8b195ee0364ee5af, State: Running, Role: LEADER
I20260926 01:09:55.717200 485 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7243178f18ce464b8b195ee0364ee5af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7243178f18ce464b8b195ee0364ee5af" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45905 } }
I20260926 01:09:55.717258 480 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7243178f18ce464b8b195ee0364ee5af [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:09:55.717514 485 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7243178f18ce464b8b195ee0364ee5af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7243178f18ce464b8b195ee0364ee5af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7243178f18ce464b8b195ee0364ee5af" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45905 } } }
I20260926 01:09:55.717660 485 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7243178f18ce464b8b195ee0364ee5af [sys.catalog]: This master's current role is: LEADER
I20260926 01:09:55.717842 485 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7243178f18ce464b8b195ee0364ee5af [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7243178f18ce464b8b195ee0364ee5af. Latest consensus state: current_term: 1 leader_uuid: "7243178f18ce464b8b195ee0364ee5af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7243178f18ce464b8b195ee0364ee5af" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45905 } } }
I20260926 01:09:55.717967 485 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7243178f18ce464b8b195ee0364ee5af [sys.catalog]: This master's current role is: LEADER
I20260926 01:09:55.718250 492 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:09:55.718765 492 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:09:55.720506 492 catalog_manager.cc:1347] Generated new cluster ID: bee06bf1565741ed93fac5112c4b132a
I20260926 01:09:55.720561 492 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 01:09:55 dist-test-slave-7b22 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384995, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:09:55.727506 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:55.712701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34256 (local address 127.0.98.62:45905)
0926 01:09:55.712894 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 01:09:55.712904 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 01:09:55.713202 (+ 298us) server_negotiation.cc:407] Connection header received
0926 01:09:55.715421 (+ 2219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:09:55.715427 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:09:55.715485 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:09:55.715649 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:09:55.717249 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:55.718666 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:55.720119 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:55.720695 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:55.720797 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:09:55.724062 (+ 3265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:09:55.724067 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:09:55.724068 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:09:55.724079 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:09:55.725645 (+ 1566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:09:55.726103 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:09:55.726106 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:09:55.726107 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:09:55.726209 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:09:55.726459 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:09:55.726459 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:09:55.726460 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:09:55.726698 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:09:55.726854 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0926 01:09:55.727123 (+ 269us) server_negotiation.cc:299] Negotiation successful
0926 01:09:55.727261 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":83,"threads_started":1}
I20260926 01:09:55.730134 492 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:09:55.730657 492 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:09:55.735090 492 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7243178f18ce464b8b195ee0364ee5af: Generated new TSK 0
I20260926 01:09:55.735277 492 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384993574762-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384993574762-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 01:09:55.765202 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384993574762-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384993574762-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384993574762-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384993574762-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45905
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384993574762-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:09:55.845774 506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:09:55.846037 506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:09:55.846063 506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:09:55.847280 506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:09:55.847324 506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:09:55.847352 506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 01:09:55.848824 506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384993574762-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384993574762-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384993574762-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384993574762-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45905
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384993574762-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:09:55.849047 506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:09:55.849291 506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:09:55.854120 511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:09:55.854112 512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:09:55.854187 514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:09:55.854233 506 server_base.cc:1034] running on GCE node
I20260926 01:09:55.854539 506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:09:55.854795 506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:09:55.855963 506 hybrid_clock.cc:648] HybridClock initialized: now 1790384995855938 us; error 35 us; skew 500 ppm
Sep 26 01:09:55 dist-test-slave-7b22 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384995, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:09:55.858526 506 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 01:09:55.858930 506 webserver.cc:460] Webserver started at http://127.0.98.1:39499/ using document root <none> and password file <none>
I20260926 01:09:55.859117 506 fs_manager.cc:359] Metadata directory not provided
I20260926 01:09:55.859157 506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:09:55.859282 506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:09:55.860340 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384993574762-392-0/minicluster-data/ts-0/data/instance:
uuid: "467f773ae0b14763afb3e59e8c6b9145"
format_stamp: "Formatted at 2026-09-26 01:09:55 on dist-test-slave-7b22"
server_key: "0eeb96f25241f48820a1346f28019f96"
server_key_iv: "09fa00236086dc074cf8df5f1fbb1836"
server_key_version: "encryptionkey@0"
I20260926 01:09:55.860647 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384993574762-392-0/minicluster-data/ts-0/wal/instance:
uuid: "467f773ae0b14763afb3e59e8c6b9145"
format_stamp: "Formatted at 2026-09-26 01:09:55 on dist-test-slave-7b22"
server_key: "0eeb96f25241f48820a1346f28019f96"
server_key_iv: "09fa00236086dc074cf8df5f1fbb1836"
server_key_version: "encryptionkey@0"
I20260926 01:09:55.861922 506 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 01:09:55.862632 521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:09:55.862773 506 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 01:09:55.862844 506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384993574762-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384993574762-392-0/minicluster-data/ts-0/wal
uuid: "467f773ae0b14763afb3e59e8c6b9145"
format_stamp: "Formatted at 2026-09-26 01:09:55 on dist-test-slave-7b22"
server_key: "0eeb96f25241f48820a1346f28019f96"
server_key_iv: "09fa00236086dc074cf8df5f1fbb1836"
server_key_version: "encryptionkey@0"
I20260926 01:09:55.862912 506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384993574762-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384993574762-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384993574762-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:09:55.874900 506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:09:55.876924 506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:09:55.877053 506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:09:55.877280 506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:09:55.877561 506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:09:55.877595 506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:09:55.877617 506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:09:55.877641 506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:09:55.883064 506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41269
I20260926 01:09:55.883131 634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41269 every 8 connection(s)
I20260926 01:09:55.883411 506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384993574762-392-0/minicluster-data/ts-0/data/info.pb
I20260926 01:09:55.889806 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 506
I20260926 01:09:55.889933 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384993574762-392-0/minicluster-data/ts-0/wal/instance
I20260926 01:09:55.890084 392 external_mini_cluster.cc:1428] Setting key 24c1bcd8786bdea20a8b1e45022bb5bc
Sep 26 01:09:55 dist-test-slave-7b22 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384995, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:09:55.893415 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:55.884115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37665 (local address 127.0.98.62:45905)
0926 01:09:55.884186 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 01:09:55.884188 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:09:55.884767 (+ 579us) server_negotiation.cc:407] Connection header received
0926 01:09:55.885260 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:09:55.885261 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:09:55.885274 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:09:55.885327 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:09:55.886352 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:55.886888 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:55.888034 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:55.888262 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:55.888334 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:09:55.890787 (+ 2453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:09:55.890789 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:09:55.890790 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:09:55.890797 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:09:55.892093 (+ 1296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:09:55.892504 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:09:55.892505 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:09:55.892506 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:09:55.892563 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:09:55.892784 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:09:55.892784 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:09:55.892785 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:09:55.892891 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:09:55.892945 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 01:09:55.893231 (+ 286us) server_negotiation.cc:299] Negotiation successful
0926 01:09:55.893323 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 01:09:55.893564 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:55.884046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45905 (local address 127.0.98.1:37665)
0926 01:09:55.884208 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 01:09:55.884246 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 01:09:55.885177 (+ 931us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:09:55.885392 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:09:55.885396 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:09:55.886042 (+ 646us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:09:55.886243 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:09:55.886248 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:09:55.886994 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:09:55.886994 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:09:55.887936 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:09:55.887939 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:09:55.888373 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:09:55.888374 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:09:55.888538 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:09:55.889150 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:09:55.889159 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:09:55.890699 (+ 1540us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:09:55.892189 (+ 1490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:09:55.892195 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:09:55.892198 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:09:55.892436 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:09:55.892624 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:09:55.892624 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:09:55.892625 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:09:55.892729 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:09:55.892945 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:09:55.892952 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:09:55.893089 (+ 137us) client_negotiation.cc:769] Sending connection context
0926 01:09:55.893183 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 01:09:55.893341 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":72,"threads_started":1}
I20260926 01:09:55.894196 635 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45905
I20260926 01:09:55.894286 635 heartbeater.cc:464] Registering TS with master...
I20260926 01:09:55.894459 635 heartbeater.cc:511] Master 127.0.98.62:45905 requested a full tablet report, sending...
I20260926 01:09:55.894958 444 ts_manager.cc:194] Registered new tserver with Master: 467f773ae0b14763afb3e59e8c6b9145 (127.0.98.1:41269)
I20260926 01:09:55.895576 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:37665
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384993574762-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384993574762-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 01:09:55.915032 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384993574762-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384993574762-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384993574762-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384993574762-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45905
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384993574762-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:09:55.997303 640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:09:55.997534 640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:09:55.997558 640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:09:55.998780 640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:09:55.998823 640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:09:55.998852 640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 01:09:56.000391 640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384993574762-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384993574762-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384993574762-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384993574762-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45905
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384993574762-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:09:56.000610 640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:09:56.000792 640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:09:56.005730 648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:09:56.005800 646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:09:56.005940 640 server_base.cc:1034] running on GCE node
W20260926 01:09:56.005833 645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:09:56.006225 640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:09:56.006472 640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:09:56.007643 640 hybrid_clock.cc:648] HybridClock initialized: now 1790384996007625 us; error 29 us; skew 500 ppm
Sep 26 01:09:56 dist-test-slave-7b22 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384996, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:09:56.010210 640 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 01:09:56.010633 640 webserver.cc:460] Webserver started at http://127.0.98.2:33087/ using document root <none> and password file <none>
I20260926 01:09:56.010825 640 fs_manager.cc:359] Metadata directory not provided
I20260926 01:09:56.010867 640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:09:56.010954 640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:09:56.011988 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384993574762-392-0/minicluster-data/ts-1/data/instance:
uuid: "09fe06254fc64f679c988ee2f9cc862e"
format_stamp: "Formatted at 2026-09-26 01:09:56 on dist-test-slave-7b22"
server_key: "a657a6bc0c97880ca01ef6ab5fb2e614"
server_key_iv: "d812e2aa2121b7585f46f17d99588bec"
server_key_version: "encryptionkey@0"
I20260926 01:09:56.012293 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384993574762-392-0/minicluster-data/ts-1/wal/instance:
uuid: "09fe06254fc64f679c988ee2f9cc862e"
format_stamp: "Formatted at 2026-09-26 01:09:56 on dist-test-slave-7b22"
server_key: "a657a6bc0c97880ca01ef6ab5fb2e614"
server_key_iv: "d812e2aa2121b7585f46f17d99588bec"
server_key_version: "encryptionkey@0"
I20260926 01:09:56.014108 640 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20260926 01:09:56.015089 655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:09:56.015285 640 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:09:56.015368 640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384993574762-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384993574762-392-0/minicluster-data/ts-1/wal
uuid: "09fe06254fc64f679c988ee2f9cc862e"
format_stamp: "Formatted at 2026-09-26 01:09:56 on dist-test-slave-7b22"
server_key: "a657a6bc0c97880ca01ef6ab5fb2e614"
server_key_iv: "d812e2aa2121b7585f46f17d99588bec"
server_key_version: "encryptionkey@0"
I20260926 01:09:56.015430 640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384993574762-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384993574762-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384993574762-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:09:56.039218 640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:09:56.041381 640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:09:56.041512 640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:09:56.041707 640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:09:56.042008 640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:09:56.042040 640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:09:56.042061 640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:09:56.042073 640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:09:56.047773 640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42403
I20260926 01:09:56.047842 768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42403 every 8 connection(s)
I20260926 01:09:56.048121 640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384993574762-392-0/minicluster-data/ts-1/data/info.pb
I20260926 01:09:56.049799 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 640
I20260926 01:09:56.049908 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384993574762-392-0/minicluster-data/ts-1/wal/instance
I20260926 01:09:56.050053 392 external_mini_cluster.cc:1428] Setting key 8c7d8c9626bda2268a34dc817598cc3e
Sep 26 01:09:56 dist-test-slave-7b22 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384996, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260926 01:09:56.057550 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:56.048811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53059 (local address 127.0.98.62:45905)
0926 01:09:56.048879 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 01:09:56.048881 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:09:56.049043 (+ 162us) server_negotiation.cc:407] Connection header received
0926 01:09:56.049531 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:09:56.049533 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:09:56.049545 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:09:56.049591 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:09:56.050714 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:56.051227 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:56.052214 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:56.052415 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:56.052480 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:09:56.054668 (+ 2188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:09:56.054670 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:09:56.054670 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:09:56.054677 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:09:56.056075 (+ 1398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:09:56.056549 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:09:56.056550 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:09:56.056551 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:09:56.056610 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:09:56.056893 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:09:56.056894 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:09:56.056894 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:09:56.057009 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:09:56.057056 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0926 01:09:56.057374 (+ 318us) server_negotiation.cc:299] Negotiation successful
0926 01:09:56.057476 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 01:09:56.057768 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:56.048778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45905 (local address 127.0.98.2:53059)
0926 01:09:56.048923 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 01:09:56.048959 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 01:09:56.049436 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:09:56.049687 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:09:56.049691 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:09:56.050367 (+ 676us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:09:56.050588 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:09:56.050594 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:09:56.051315 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:09:56.051316 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:09:56.052128 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:09:56.052131 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:09:56.052483 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:09:56.052484 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:09:56.052611 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:09:56.053080 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:09:56.053089 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:09:56.054567 (+ 1478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:09:56.056158 (+ 1591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:09:56.056166 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:09:56.056168 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:09:56.056448 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:09:56.056674 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:09:56.056674 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:09:56.056675 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:09:56.056820 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:09:56.057065 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:09:56.057071 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:09:56.057235 (+ 164us) client_negotiation.cc:769] Sending connection context
0926 01:09:56.057352 (+ 117us) client_negotiation.cc:240] Negotiation successful
0926 01:09:56.057529 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":67,"threads_started":1}
I20260926 01:09:56.058314 769 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45905
I20260926 01:09:56.058399 769 heartbeater.cc:464] Registering TS with master...
I20260926 01:09:56.058550 769 heartbeater.cc:511] Master 127.0.98.62:45905 requested a full tablet report, sending...
I20260926 01:09:56.058887 444 ts_manager.cc:194] Registered new tserver with Master: 09fe06254fc64f679c988ee2f9cc862e (127.0.98.2:42403)
I20260926 01:09:56.059283 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:53059
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384993574762-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384993574762-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 01:09:56.076608 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384993574762-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384993574762-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384993574762-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384993574762-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45905
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384993574762-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:09:56.159709 774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:09:56.159945 774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:09:56.159968 774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:09:56.161167 774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:09:56.161211 774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:09:56.161238 774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 01:09:56.162703 774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384993574762-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384993574762-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384993574762-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384993574762-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45905
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384993574762-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:09:56.162896 774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:09:56.163080 774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:09:56.167870 780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:09:56.167943 782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:09:56.167977 774 server_base.cc:1034] running on GCE node
W20260926 01:09:56.167950 779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:09:56.168294 774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:09:56.168499 774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:09:56.169639 774 hybrid_clock.cc:648] HybridClock initialized: now 1790384996169622 us; error 33 us; skew 500 ppm
Sep 26 01:09:56 dist-test-slave-7b22 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384996, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:09:56.172086 774 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 01:09:56.172515 774 webserver.cc:460] Webserver started at http://127.0.98.3:34597/ using document root <none> and password file <none>
I20260926 01:09:56.172715 774 fs_manager.cc:359] Metadata directory not provided
I20260926 01:09:56.172761 774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:09:56.172880 774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:09:56.173993 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384993574762-392-0/minicluster-data/ts-2/data/instance:
uuid: "df607ffab9084db999cb408e51d6cfe7"
format_stamp: "Formatted at 2026-09-26 01:09:56 on dist-test-slave-7b22"
server_key: "34e1fec56a2d3871d83be4ce2109bf02"
server_key_iv: "28fde176edef439d8bb04e9a6866aa49"
server_key_version: "encryptionkey@0"
I20260926 01:09:56.174333 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384993574762-392-0/minicluster-data/ts-2/wal/instance:
uuid: "df607ffab9084db999cb408e51d6cfe7"
format_stamp: "Formatted at 2026-09-26 01:09:56 on dist-test-slave-7b22"
server_key: "34e1fec56a2d3871d83be4ce2109bf02"
server_key_iv: "28fde176edef439d8bb04e9a6866aa49"
server_key_version: "encryptionkey@0"
I20260926 01:09:56.175629 774 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:09:56.176416 789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:09:56.176580 774 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:09:56.176667 774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384993574762-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384993574762-392-0/minicluster-data/ts-2/wal
uuid: "df607ffab9084db999cb408e51d6cfe7"
format_stamp: "Formatted at 2026-09-26 01:09:56 on dist-test-slave-7b22"
server_key: "34e1fec56a2d3871d83be4ce2109bf02"
server_key_iv: "28fde176edef439d8bb04e9a6866aa49"
server_key_version: "encryptionkey@0"
I20260926 01:09:56.176730 774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384993574762-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384993574762-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384993574762-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:09:56.217767 774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:09:56.220201 774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:09:56.220343 774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:09:56.220577 774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:09:56.220886 774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:09:56.220919 774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:09:56.220949 774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:09:56.220973 774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:09:56.227228 774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41701
I20260926 01:09:56.227273 902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41701 every 8 connection(s)
I20260926 01:09:56.227586 774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384993574762-392-0/minicluster-data/ts-2/data/info.pb
I20260926 01:09:56.231743 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 774
I20260926 01:09:56.231827 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384993574762-392-0/minicluster-data/ts-2/wal/instance
I20260926 01:09:56.231966 392 external_mini_cluster.cc:1428] Setting key 1ecbd4ef4007125bf211cee40b239528
Sep 26 01:09:56 dist-test-slave-7b22 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384996, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:09:56.237340 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:56.228344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35207 (local address 127.0.98.62:45905)
0926 01:09:56.228398 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 01:09:56.228400 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:09:56.228590 (+ 190us) server_negotiation.cc:407] Connection header received
0926 01:09:56.229175 (+ 585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:09:56.229177 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:09:56.229192 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:09:56.229257 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:09:56.230453 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:56.230993 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:56.232021 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:56.232254 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:56.232318 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:09:56.234609 (+ 2291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:09:56.234610 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:09:56.234611 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:09:56.234633 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:09:56.236018 (+ 1385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:09:56.236440 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:09:56.236442 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:09:56.236442 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:09:56.236501 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:09:56.236754 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:09:56.236754 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:09:56.236755 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:09:56.236870 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:09:56.236920 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 01:09:56.237192 (+ 272us) server_negotiation.cc:299] Negotiation successful
0926 01:09:56.237272 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 01:09:56.237483 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:56.228303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45905 (local address 127.0.98.3:35207)
0926 01:09:56.228474 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0926 01:09:56.228511 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 01:09:56.229022 (+ 511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:09:56.229359 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:09:56.229364 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:09:56.230134 (+ 770us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:09:56.230361 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:09:56.230367 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:09:56.231082 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:09:56.231083 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:09:56.231944 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:09:56.231947 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:09:56.232335 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:09:56.232336 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:09:56.232497 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:09:56.233053 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:09:56.233064 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:09:56.234526 (+ 1462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:09:56.236105 (+ 1579us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:09:56.236112 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:09:56.236114 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:09:56.236356 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:09:56.236601 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:09:56.236601 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:09:56.236601 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:09:56.236705 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:09:56.236929 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:09:56.236933 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:09:56.237068 (+ 135us) client_negotiation.cc:769] Sending connection context
0926 01:09:56.237154 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 01:09:56.237285 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":75,"threads_started":1}
I20260926 01:09:56.237967 903 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45905
I20260926 01:09:56.238062 903 heartbeater.cc:464] Registering TS with master...
I20260926 01:09:56.238233 903 heartbeater.cc:511] Master 127.0.98.62:45905 requested a full tablet report, sending...
I20260926 01:09:56.238612 444 ts_manager.cc:194] Registered new tserver with Master: df607ffab9084db999cb408e51d6cfe7 (127.0.98.3:41701)
I20260926 01:09:56.238971 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:35207
I20260926 01:09:56.244026 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 01:09:56 dist-test-slave-7b22 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384996, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 01:09:56 dist-test-slave-7b22 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384996, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 01:09:56.267277 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:56.259424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41434 (local address 127.0.98.1:41269)
0926 01:09:56.259584 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0926 01:09:56.259589 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 01:09:56.259600 (+ 11us) server_negotiation.cc:407] Connection header received
0926 01:09:56.259670 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:09:56.259672 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:09:56.259687 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:09:56.259751 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:09:56.260295 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:56.261084 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:56.261904 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:56.262248 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:56.262308 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:09:56.264606 (+ 2298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:09:56.264615 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:09:56.264615 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:09:56.264622 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:09:56.266179 (+ 1557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:09:56.266536 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:09:56.266539 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:09:56.266540 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:09:56.266610 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:09:56.266801 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:09:56.266802 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:09:56.266802 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:09:56.266955 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:09:56.267082 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0926 01:09:56.267119 (+ 37us) server_negotiation.cc:299] Negotiation successful
0926 01:09:56.267188 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":77,"threads_started":1}
W20260926 01:09:56.267638 548 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41434
Sep 26 01:09:56 dist-test-slave-7b22 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384996, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 01:09:56 dist-test-slave-7b22 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384996, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 01:09:56.288618 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:56.281719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41440 (local address 127.0.98.1:41269)
0926 01:09:56.281777 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 01:09:56.281779 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:09:56.281883 (+ 104us) server_negotiation.cc:407] Connection header received
0926 01:09:56.281975 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:09:56.281976 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:09:56.281987 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:09:56.282028 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:09:56.282588 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:56.283073 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:56.283782 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:56.283982 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:56.284035 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:09:56.286362 (+ 2327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:09:56.286363 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:09:56.286364 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:09:56.286373 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:09:56.287548 (+ 1175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:09:56.287894 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:09:56.287895 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:09:56.287896 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:09:56.287959 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:09:56.288131 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:09:56.288132 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:09:56.288133 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:09:56.288244 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:09:56.288349 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 01:09:56.288428 (+ 79us) server_negotiation.cc:299] Negotiation successful
0926 01:09:56.288511 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 01:09:56.289204 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 506
I20260926 01:09:56.293457 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 640
I20260926 01:09:56.298413 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 774
I20260926 01:09:56.302132 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 412
2026-09-26T01:09:56Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2730 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384993574762-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 01:09:56 dist-test-slave-7b22 krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 01:09:56 dist-test-slave-7b22 krb5kdc[917](info): set up 2 sockets
Sep 26 01:09:56 dist-test-slave-7b22 krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260926 01:09:57.257179 392 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384993574762-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384993574762-392-0/krb5kdc/test-user.keytab.
Sep 26 01:09:57 dist-test-slave-7b22 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384997, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T01:09:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:09:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384993574762-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384993574762-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 01:09:57.373670 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384993574762-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384993574762-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384993574762-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42625
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384993574762-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42625
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:09:57.453325 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:09:57.453540 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:09:57.453562 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:09:57.454697 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:09:57.454735 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:09:57.454748 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:09:57.454758 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:09:57.454769 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:09:57.456183 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384993574762-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384993574762-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42625
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384993574762-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42625
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384993574762-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:09:57.456380 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:09:57.456557 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:09:57.461166 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:09:57.461174 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:09:57.461350 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:09:57.461288 930 server_base.cc:1034] running on GCE node
I20260926 01:09:57.461619 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:09:57.461930 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:09:57.463104 930 hybrid_clock.cc:648] HybridClock initialized: now 1790384997463095 us; error 40 us; skew 500 ppm
Sep 26 01:09:57 dist-test-slave-7b22 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384997, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:09:57.465520 930 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 01:09:57.465987 930 webserver.cc:460] Webserver started at http://127.0.98.62:44151/ using document root <none> and password file <none>
I20260926 01:09:57.466215 930 fs_manager.cc:359] Metadata directory not provided
I20260926 01:09:57.466262 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:09:57.466388 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:09:57.467410 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384993574762-392-0/minicluster-data/master-0/data/instance:
uuid: "b77bbea8e2904527a00604fbd93c4694"
format_stamp: "Formatted at 2026-09-26 01:09:57 on dist-test-slave-7b22"
server_key: "160eb9b5dc2f1b664354eb4aaa4647e3"
server_key_iv: "adb394687abd527919e553699523856d"
server_key_version: "encryptionkey@0"
I20260926 01:09:57.467747 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384993574762-392-0/minicluster-data/master-0/wal/instance:
uuid: "b77bbea8e2904527a00604fbd93c4694"
format_stamp: "Formatted at 2026-09-26 01:09:57 on dist-test-slave-7b22"
server_key: "160eb9b5dc2f1b664354eb4aaa4647e3"
server_key_iv: "adb394687abd527919e553699523856d"
server_key_version: "encryptionkey@0"
I20260926 01:09:57.469010 930 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 01:09:57.469624 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:09:57.469781 930 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 01:09:57.469887 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384993574762-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384993574762-392-0/minicluster-data/master-0/wal
uuid: "b77bbea8e2904527a00604fbd93c4694"
format_stamp: "Formatted at 2026-09-26 01:09:57 on dist-test-slave-7b22"
server_key: "160eb9b5dc2f1b664354eb4aaa4647e3"
server_key_iv: "adb394687abd527919e553699523856d"
server_key_version: "encryptionkey@0"
I20260926 01:09:57.469957 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384993574762-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384993574762-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384993574762-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:09:57.477010 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:09:57.479236 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:09:57.479372 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:09:57.483096 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42625
I20260926 01:09:57.483129 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42625 every 8 connection(s)
I20260926 01:09:57.483451 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
I20260926 01:09:57.484280 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:09:57.486613 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b77bbea8e2904527a00604fbd93c4694: Bootstrap starting.
I20260926 01:09:57.487164 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b77bbea8e2904527a00604fbd93c4694: Neither blocks nor log segments found. Creating new log.
I20260926 01:09:57.487423 998 log.cc:824] T 00000000000000000000000000000000 P b77bbea8e2904527a00604fbd93c4694: Log is configured to *not* fsync() on all Append() calls
I20260926 01:09:57.488099 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b77bbea8e2904527a00604fbd93c4694: No bootstrap required, opened a new log
I20260926 01:09:57.488132 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 930
I20260926 01:09:57.488224 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384993574762-392-0/minicluster-data/master-0/wal/instance
I20260926 01:09:57.488376 392 external_mini_cluster.cc:1428] Setting key 3c24939ff605314c697ec160806c6dc9
I20260926 01:09:57.489921 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P b77bbea8e2904527a00604fbd93c4694 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b77bbea8e2904527a00604fbd93c4694" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42625 } }
I20260926 01:09:57.490064 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P b77bbea8e2904527a00604fbd93c4694 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:09:57.490097 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P b77bbea8e2904527a00604fbd93c4694 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b77bbea8e2904527a00604fbd93c4694, State: Initialized, Role: FOLLOWER
I20260926 01:09:57.490236 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P b77bbea8e2904527a00604fbd93c4694 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b77bbea8e2904527a00604fbd93c4694" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42625 } }
I20260926 01:09:57.490300 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P b77bbea8e2904527a00604fbd93c4694 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:09:57.490329 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P b77bbea8e2904527a00604fbd93c4694 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:09:57.490374 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b77bbea8e2904527a00604fbd93c4694 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:09:57.491209 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P b77bbea8e2904527a00604fbd93c4694 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b77bbea8e2904527a00604fbd93c4694" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42625 } }
I20260926 01:09:57.491304 998 leader_election.cc:302] T 00000000000000000000000000000000 P b77bbea8e2904527a00604fbd93c4694 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b77bbea8e2904527a00604fbd93c4694; no voters:
I20260926 01:09:57.491459 998 leader_election.cc:288] T 00000000000000000000000000000000 P b77bbea8e2904527a00604fbd93c4694 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:09:57.491497 1003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b77bbea8e2904527a00604fbd93c4694 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:09:57.491601 1003 raft_consensus.cc:686] T 00000000000000000000000000000000 P b77bbea8e2904527a00604fbd93c4694 [term 1 LEADER]: Becoming Leader. State: Replica: b77bbea8e2904527a00604fbd93c4694, State: Running, Role: LEADER
I20260926 01:09:57.491680 1003 consensus_queue.cc:237] T 00000000000000000000000000000000 P b77bbea8e2904527a00604fbd93c4694 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b77bbea8e2904527a00604fbd93c4694" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42625 } }
I20260926 01:09:57.491719 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P b77bbea8e2904527a00604fbd93c4694 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:09:57.492210 1003 sys_catalog.cc:455] T 00000000000000000000000000000000 P b77bbea8e2904527a00604fbd93c4694 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b77bbea8e2904527a00604fbd93c4694. Latest consensus state: current_term: 1 leader_uuid: "b77bbea8e2904527a00604fbd93c4694" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b77bbea8e2904527a00604fbd93c4694" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42625 } } }
I20260926 01:09:57.492216 1005 sys_catalog.cc:455] T 00000000000000000000000000000000 P b77bbea8e2904527a00604fbd93c4694 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b77bbea8e2904527a00604fbd93c4694" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b77bbea8e2904527a00604fbd93c4694" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42625 } } }
I20260926 01:09:57.492391 1005 sys_catalog.cc:458] T 00000000000000000000000000000000 P b77bbea8e2904527a00604fbd93c4694 [sys.catalog]: This master's current role is: LEADER
I20260926 01:09:57.492592 1003 sys_catalog.cc:458] T 00000000000000000000000000000000 P b77bbea8e2904527a00604fbd93c4694 [sys.catalog]: This master's current role is: LEADER
I20260926 01:09:57.492936 1010 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:09:57.493438 1010 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:09:57.495242 1010 catalog_manager.cc:1347] Generated new cluster ID: 7853a3a5e7024931bea0fd075bd9640b
I20260926 01:09:57.495292 1010 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 01:09:57 dist-test-slave-7b22 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384997, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:09:57.500607 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:57.488887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42260 (local address 127.0.98.62:42625)
0926 01:09:57.489059 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 01:09:57.489069 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 01:09:57.489110 (+ 41us) server_negotiation.cc:407] Connection header received
0926 01:09:57.489917 (+ 807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:09:57.489926 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:09:57.489999 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:09:57.490157 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:09:57.490899 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:57.492181 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:57.493303 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:57.493912 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:57.494025 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:09:57.496742 (+ 2717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:09:57.496747 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:09:57.496747 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:09:57.496758 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:09:57.498637 (+ 1879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:09:57.499066 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:09:57.499069 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:09:57.499070 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:09:57.499181 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:09:57.499387 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:09:57.499388 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:09:57.499388 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:09:57.499684 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:09:57.500205 (+ 521us) server_negotiation.cc:1035] Waiting for connection context
0926 01:09:57.500269 (+ 64us) server_negotiation.cc:299] Negotiation successful
0926 01:09:57.500401 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":58,"threads_started":1}
I20260926 01:09:57.516330 1010 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:09:57.516857 1010 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:09:57.521214 1010 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b77bbea8e2904527a00604fbd93c4694: Generated new TSK 0
I20260926 01:09:57.521389 1010 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384993574762-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384993574762-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 01:09:57.549039 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384993574762-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384993574762-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384993574762-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384993574762-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42625
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384993574762-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:09:57.630355 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:09:57.630575 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:09:57.630599 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:09:57.631771 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:09:57.631815 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:09:57.631841 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 01:09:57.633291 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384993574762-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384993574762-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384993574762-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384993574762-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42625
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384993574762-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:09:57.633523 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:09:57.633734 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:09:57.638685 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:09:57.638756 1024 server_base.cc:1034] running on GCE node
W20260926 01:09:57.638689 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:09:57.638688 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:09:57.639110 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:09:57.639344 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:09:57.640502 1024 hybrid_clock.cc:648] HybridClock initialized: now 1790384997640483 us; error 39 us; skew 500 ppm
Sep 26 01:09:57 dist-test-slave-7b22 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384997, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:09:57.643162 1024 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 01:09:57.643591 1024 webserver.cc:460] Webserver started at http://127.0.98.1:46863/ using document root <none> and password file <none>
I20260926 01:09:57.643800 1024 fs_manager.cc:359] Metadata directory not provided
I20260926 01:09:57.643848 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:09:57.643970 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:09:57.645056 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384993574762-392-0/minicluster-data/ts-0/data/instance:
uuid: "4b7e8d481c9a4588bd273bad8696c41f"
format_stamp: "Formatted at 2026-09-26 01:09:57 on dist-test-slave-7b22"
server_key: "0fd378814cb972a0d90020ba5f70e8de"
server_key_iv: "c2a01aa2b07d86a542ff1325deeae7da"
server_key_version: "encryptionkey@0"
I20260926 01:09:57.645386 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384993574762-392-0/minicluster-data/ts-0/wal/instance:
uuid: "4b7e8d481c9a4588bd273bad8696c41f"
format_stamp: "Formatted at 2026-09-26 01:09:57 on dist-test-slave-7b22"
server_key: "0fd378814cb972a0d90020ba5f70e8de"
server_key_iv: "c2a01aa2b07d86a542ff1325deeae7da"
server_key_version: "encryptionkey@0"
I20260926 01:09:57.646703 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260926 01:09:57.647383 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:09:57.647552 1024 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 01:09:57.647653 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384993574762-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384993574762-392-0/minicluster-data/ts-0/wal
uuid: "4b7e8d481c9a4588bd273bad8696c41f"
format_stamp: "Formatted at 2026-09-26 01:09:57 on dist-test-slave-7b22"
server_key: "0fd378814cb972a0d90020ba5f70e8de"
server_key_iv: "c2a01aa2b07d86a542ff1325deeae7da"
server_key_version: "encryptionkey@0"
I20260926 01:09:57.647713 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384993574762-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384993574762-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384993574762-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:09:57.661315 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:09:57.663491 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:09:57.663682 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:09:57.663911 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:09:57.664193 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:09:57.664225 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:09:57.664253 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:09:57.664278 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:09:57.669806 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46415
I20260926 01:09:57.669914 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46415 every 8 connection(s)
I20260926 01:09:57.670189 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384993574762-392-0/minicluster-data/ts-0/data/info.pb
I20260926 01:09:57.673597 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 1024
I20260926 01:09:57.673707 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384993574762-392-0/minicluster-data/ts-0/wal/instance
I20260926 01:09:57.673899 392 external_mini_cluster.cc:1428] Setting key 25f952ab6693588af32a0a90755ac2f4
Sep 26 01:09:57 dist-test-slave-7b22 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384997, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:09:57.679967 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:57.670926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42933 (local address 127.0.98.62:42625)
0926 01:09:57.671003 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0926 01:09:57.671005 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:09:57.671208 (+ 203us) server_negotiation.cc:407] Connection header received
0926 01:09:57.671679 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:09:57.671680 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:09:57.671693 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:09:57.671754 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:09:57.672730 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:57.673236 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:57.674525 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:57.674745 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:57.674794 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:09:57.677244 (+ 2450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:09:57.677246 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:09:57.677246 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:09:57.677259 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:09:57.678684 (+ 1425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:09:57.679060 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:09:57.679062 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:09:57.679062 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:09:57.679120 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:09:57.679331 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:09:57.679332 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:09:57.679333 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:09:57.679460 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:09:57.679516 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 01:09:57.679784 (+ 268us) server_negotiation.cc:299] Negotiation successful
0926 01:09:57.679875 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 01:09:57.680135 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:57.670857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42625 (local address 127.0.98.1:42933)
0926 01:09:57.671108 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0926 01:09:57.671139 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0926 01:09:57.671601 (+ 462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:09:57.671812 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:09:57.671815 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:09:57.672441 (+ 626us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:09:57.672644 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:09:57.672650 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:09:57.673307 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:09:57.673308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:09:57.674442 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:09:57.674445 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:09:57.674819 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:09:57.674821 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:09:57.674968 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:09:57.675497 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:09:57.675508 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:09:57.677147 (+ 1639us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:09:57.678762 (+ 1615us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:09:57.678769 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:09:57.678771 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:09:57.678988 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:09:57.679170 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:09:57.679171 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:09:57.679171 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:09:57.679278 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:09:57.679514 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:09:57.679520 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:09:57.679666 (+ 146us) client_negotiation.cc:769] Sending connection context
0926 01:09:57.679755 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 01:09:57.679915 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":75,"threads_started":1}
I20260926 01:09:57.680577 1153 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42625
I20260926 01:09:57.680668 1153 heartbeater.cc:464] Registering TS with master...
I20260926 01:09:57.680840 1153 heartbeater.cc:511] Master 127.0.98.62:42625 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260926 01:09:57.681355 962 ts_manager.cc:194] Registered new tserver with Master: 4b7e8d481c9a4588bd273bad8696c41f (127.0.98.1:46415)
I20260926 01:09:57.682024 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:42933
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384993574762-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384993574762-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 01:09:57.701865 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384993574762-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384993574762-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384993574762-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384993574762-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42625
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384993574762-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:09:57.783690 1158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:09:57.783914 1158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:09:57.783941 1158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:09:57.785089 1158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:09:57.785125 1158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:09:57.785151 1158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 01:09:57.786616 1158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384993574762-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384993574762-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384993574762-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384993574762-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42625
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384993574762-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:09:57.786808 1158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:09:57.786993 1158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:09:57.791923 1164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:09:57.791985 1163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:09:57.791913 1166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:09:57.792011 1158 server_base.cc:1034] running on GCE node
I20260926 01:09:57.792279 1158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:09:57.792516 1158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:09:57.793686 1158 hybrid_clock.cc:648] HybridClock initialized: now 1790384997793674 us; error 35 us; skew 500 ppm
Sep 26 01:09:57 dist-test-slave-7b22 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384997, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:09:57.796114 1158 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 01:09:57.796553 1158 webserver.cc:460] Webserver started at http://127.0.98.2:34387/ using document root <none> and password file <none>
I20260926 01:09:57.796759 1158 fs_manager.cc:359] Metadata directory not provided
I20260926 01:09:57.796806 1158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:09:57.796931 1158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:09:57.798055 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384993574762-392-0/minicluster-data/ts-1/data/instance:
uuid: "2c688214a3c043e993f2266ece210fb2"
format_stamp: "Formatted at 2026-09-26 01:09:57 on dist-test-slave-7b22"
server_key: "756675d90e65ec68ae4ae528244d5b4d"
server_key_iv: "183e199c8ed5b5e9d77db0eb74424bac"
server_key_version: "encryptionkey@0"
I20260926 01:09:57.798390 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384993574762-392-0/minicluster-data/ts-1/wal/instance:
uuid: "2c688214a3c043e993f2266ece210fb2"
format_stamp: "Formatted at 2026-09-26 01:09:57 on dist-test-slave-7b22"
server_key: "756675d90e65ec68ae4ae528244d5b4d"
server_key_iv: "183e199c8ed5b5e9d77db0eb74424bac"
server_key_version: "encryptionkey@0"
I20260926 01:09:57.799736 1158 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260926 01:09:57.800433 1173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:09:57.800580 1158 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 01:09:57.800668 1158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384993574762-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384993574762-392-0/minicluster-data/ts-1/wal
uuid: "2c688214a3c043e993f2266ece210fb2"
format_stamp: "Formatted at 2026-09-26 01:09:57 on dist-test-slave-7b22"
server_key: "756675d90e65ec68ae4ae528244d5b4d"
server_key_iv: "183e199c8ed5b5e9d77db0eb74424bac"
server_key_version: "encryptionkey@0"
I20260926 01:09:57.800733 1158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384993574762-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384993574762-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384993574762-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:09:57.811154 1158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:09:57.813133 1158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:09:57.813270 1158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:09:57.813493 1158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:09:57.813788 1158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:09:57.813838 1158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:09:57.813869 1158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:09:57.813894 1158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:09:57.819350 1158 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33981
I20260926 01:09:57.819389 1286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33981 every 8 connection(s)
I20260926 01:09:57.819701 1158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384993574762-392-0/minicluster-data/ts-1/data/info.pb
Sep 26 01:09:57 dist-test-slave-7b22 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384997, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:09:57.826508 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 1158
I20260926 01:09:57.826635 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384993574762-392-0/minicluster-data/ts-1/wal/instance
I20260926 01:09:57.826805 392 external_mini_cluster.cc:1428] Setting key 5f4c5ff3244fc6428460cf020e677167
I20260926 01:09:57.828578 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:57.820423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51987 (local address 127.0.98.62:42625)
0926 01:09:57.820482 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 01:09:57.820484 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:09:57.820675 (+ 191us) server_negotiation.cc:407] Connection header received
0926 01:09:57.821169 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:09:57.821170 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:09:57.821179 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:09:57.821232 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:09:57.822251 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:57.822765 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:57.823713 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:57.823891 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:57.823934 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:09:57.825914 (+ 1980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:09:57.825916 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:09:57.825916 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:09:57.825923 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:09:57.827272 (+ 1349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:09:57.827697 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:09:57.827698 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:09:57.827699 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:09:57.827755 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:09:57.827958 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:09:57.827959 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:09:57.827959 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:09:57.828098 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:09:57.828158 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 01:09:57.828411 (+ 253us) server_negotiation.cc:299] Negotiation successful
0926 01:09:57.828489 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 01:09:57.828883 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:57.820395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42625 (local address 127.0.98.2:51987)
0926 01:09:57.820543 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 01:09:57.820588 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 01:09:57.821075 (+ 487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:09:57.821295 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:09:57.821298 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:09:57.821949 (+ 651us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:09:57.822155 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:09:57.822160 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:09:57.822838 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:09:57.822839 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:09:57.823652 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:09:57.823654 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:09:57.823942 (+ 288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:09:57.823942 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:09:57.824034 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:09:57.824523 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:09:57.824530 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:09:57.825792 (+ 1262us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:09:57.827338 (+ 1546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:09:57.827345 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:09:57.827347 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:09:57.827602 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:09:57.827812 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:09:57.827812 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:09:57.827812 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:09:57.827910 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:09:57.828159 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:09:57.828164 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:09:57.828298 (+ 134us) client_negotiation.cc:769] Sending connection context
0926 01:09:57.828383 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 01:09:57.828505 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":67,"threads_started":1}
I20260926 01:09:57.829345 1287 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42625
I20260926 01:09:57.829443 1287 heartbeater.cc:464] Registering TS with master...
I20260926 01:09:57.829633 1287 heartbeater.cc:511] Master 127.0.98.62:42625 requested a full tablet report, sending...
I20260926 01:09:57.830054 962 ts_manager.cc:194] Registered new tserver with Master: 2c688214a3c043e993f2266ece210fb2 (127.0.98.2:33981)
I20260926 01:09:57.830406 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:51987
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384993574762-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384993574762-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 01:09:57.852815 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384993574762-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384993574762-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384993574762-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384993574762-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42625
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384993574762-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:09:57.932687 1292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:09:57.932905 1292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:09:57.932938 1292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:09:57.934154 1292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:09:57.934195 1292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:09:57.934230 1292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 01:09:57.935709 1292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384993574762-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384993574762-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384993574762-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384993574762-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42625
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384993574762-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:09:57.935977 1292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:09:57.936239 1292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:09:57.941260 1297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:09:57.941278 1300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:09:57.941425 1298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:09:57.941588 1292 server_base.cc:1034] running on GCE node
I20260926 01:09:57.941793 1292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:09:57.942109 1292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:09:57.943277 1292 hybrid_clock.cc:648] HybridClock initialized: now 1790384997943267 us; error 34 us; skew 500 ppm
Sep 26 01:09:57 dist-test-slave-7b22 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384997, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:09:57.945847 1292 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 01:09:57.946275 1292 webserver.cc:460] Webserver started at http://127.0.98.3:33657/ using document root <none> and password file <none>
I20260926 01:09:57.946491 1292 fs_manager.cc:359] Metadata directory not provided
I20260926 01:09:57.946539 1292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:09:57.946666 1292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:09:57.947691 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384993574762-392-0/minicluster-data/ts-2/data/instance:
uuid: "47e09b1a8ead447ab88345139de3526a"
format_stamp: "Formatted at 2026-09-26 01:09:57 on dist-test-slave-7b22"
server_key: "519e5643847ffad693e6379226a580b6"
server_key_iv: "daec1bd889643b2217afd88c3a76f89a"
server_key_version: "encryptionkey@0"
I20260926 01:09:57.948021 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384993574762-392-0/minicluster-data/ts-2/wal/instance:
uuid: "47e09b1a8ead447ab88345139de3526a"
format_stamp: "Formatted at 2026-09-26 01:09:57 on dist-test-slave-7b22"
server_key: "519e5643847ffad693e6379226a580b6"
server_key_iv: "daec1bd889643b2217afd88c3a76f89a"
server_key_version: "encryptionkey@0"
I20260926 01:09:57.949309 1292 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 01:09:57.950107 1307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:09:57.950253 1292 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:09:57.950332 1292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384993574762-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384993574762-392-0/minicluster-data/ts-2/wal
uuid: "47e09b1a8ead447ab88345139de3526a"
format_stamp: "Formatted at 2026-09-26 01:09:57 on dist-test-slave-7b22"
server_key: "519e5643847ffad693e6379226a580b6"
server_key_iv: "daec1bd889643b2217afd88c3a76f89a"
server_key_version: "encryptionkey@0"
I20260926 01:09:57.950390 1292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384993574762-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384993574762-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384993574762-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:09:57.959645 1292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:09:57.961638 1292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:09:57.961769 1292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:09:57.962001 1292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:09:57.962280 1292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:09:57.962311 1292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:09:57.962340 1292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:09:57.962368 1292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:09:57.967813 1292 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41149
I20260926 01:09:57.967855 1420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41149 every 8 connection(s)
I20260926 01:09:57.968158 1292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384993574762-392-0/minicluster-data/ts-2/data/info.pb
Sep 26 01:09:57 dist-test-slave-7b22 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384997, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:09:57.977174 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:57.968849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39871 (local address 127.0.98.62:42625)
0926 01:09:57.968914 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 01:09:57.968916 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:09:57.969102 (+ 186us) server_negotiation.cc:407] Connection header received
0926 01:09:57.969639 (+ 537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:09:57.969640 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:09:57.969652 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:09:57.969698 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:09:57.970657 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:57.971151 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:57.972107 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:57.972325 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:57.972379 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:09:57.974492 (+ 2113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:09:57.974494 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:09:57.974494 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:09:57.974503 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:09:57.975840 (+ 1337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:09:57.976247 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:09:57.976248 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:09:57.976248 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:09:57.976306 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:09:57.976521 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:09:57.976521 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:09:57.976522 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:09:57.976676 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:09:57.976734 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 01:09:57.976996 (+ 262us) server_negotiation.cc:299] Negotiation successful
0926 01:09:57.977076 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 01:09:57.977363 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:57.968816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42625 (local address 127.0.98.3:39871)
0926 01:09:57.968987 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0926 01:09:57.969023 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 01:09:57.969525 (+ 502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:09:57.969782 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:09:57.969786 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:09:57.970370 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:09:57.970568 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:09:57.970573 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:09:57.971227 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:09:57.971228 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:09:57.972029 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:09:57.972031 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:09:57.972396 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:09:57.972397 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:09:57.972525 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:09:57.972963 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:09:57.972971 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:09:57.974389 (+ 1418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:09:57.975929 (+ 1540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:09:57.975936 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:09:57.975939 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:09:57.976177 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:09:57.976359 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:09:57.976360 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:09:57.976360 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:09:57.976465 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:09:57.976736 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:09:57.976743 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:09:57.976877 (+ 134us) client_negotiation.cc:769] Sending connection context
0926 01:09:57.976966 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 01:09:57.977143 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":89,"threads_started":1}
I20260926 01:09:57.977603 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 1292
I20260926 01:09:57.977708 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384993574762-392-0/minicluster-data/ts-2/wal/instance
I20260926 01:09:57.977859 1421 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42625
I20260926 01:09:57.977916 392 external_mini_cluster.cc:1428] Setting key 7bb47c69ae55d0fcb9cc1db80c8faa9c
I20260926 01:09:57.977957 1421 heartbeater.cc:464] Registering TS with master...
I20260926 01:09:57.978135 1421 heartbeater.cc:511] Master 127.0.98.62:42625 requested a full tablet report, sending...
I20260926 01:09:57.978482 962 ts_manager.cc:194] Registered new tserver with Master: 47e09b1a8ead447ab88345139de3526a (127.0.98.3:41149)
I20260926 01:09:57.978731 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 01:09:57.978880 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:39871
I20260926 01:09:57.986091 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:57.980261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42270 (local address 127.0.98.62:42625)
0926 01:09:57.980303 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 01:09:57.980305 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:09:57.980419 (+ 114us) server_negotiation.cc:407] Connection header received
0926 01:09:57.980470 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:09:57.980471 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:09:57.980480 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:09:57.980587 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:09:57.981115 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:57.981589 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:57.982349 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:57.982553 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:57.982605 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:09:57.983756 (+ 1151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:09:57.983758 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:09:57.983758 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:09:57.983765 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:09:57.984970 (+ 1205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:09:57.985276 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:09:57.985277 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:09:57.985277 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:09:57.985330 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:09:57.985487 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:09:57.985488 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:09:57.985488 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:09:57.985645 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:09:57.985736 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 01:09:57.985892 (+ 156us) server_negotiation.cc:299] Negotiation successful
0926 01:09:57.985978 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 01:09:57.988083 962 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42270:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:09:57.989607 962 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:09:57.995682 1429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:57.991652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41149 (local address 127.0.0.1:59722)
0926 01:09:57.991975 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0926 01:09:57.991982 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0926 01:09:57.992025 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:09:57.992272 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:09:57.992275 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:09:57.992279 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:09:57.992349 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:09:57.992351 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:09:57.993657 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:09:57.993658 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:09:57.994611 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:09:57.994614 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:09:57.995389 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:09:57.995390 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:09:57.995463 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:09:57.995476 (+ 13us) client_negotiation.cc:769] Sending connection context
0926 01:09:57.995547 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 01:09:57.995601 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"mutex_wait_us":78,"thread_start_us":79,"threads_started":1}
I20260926 01:09:57.995688 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:57.991744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59722 (local address 127.0.98.3:41149)
0926 01:09:57.991915 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 01:09:57.991920 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 01:09:57.992088 (+ 168us) server_negotiation.cc:407] Connection header received
0926 01:09:57.992173 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:09:57.992175 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:09:57.992191 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:09:57.992254 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:09:57.992410 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:57.993493 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:57.994690 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:57.995318 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:57.995379 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:09:57.995437 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 01:09:57.995564 (+ 127us) server_negotiation.cc:299] Negotiation successful
0926 01:09:57.995602 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":84,"threads_started":1}
I20260926 01:09:57.996160 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:57.992078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43374 (local address 127.0.98.1:46415)
0926 01:09:57.992281 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 01:09:57.992286 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 01:09:57.992295 (+ 9us) server_negotiation.cc:407] Connection header received
0926 01:09:57.992328 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:09:57.992330 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:09:57.992344 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:09:57.992426 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:09:57.992546 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:57.993518 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:57.995027 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:57.995737 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:57.995802 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:09:57.995858 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 01:09:57.996034 (+ 176us) server_negotiation.cc:299] Negotiation successful
0926 01:09:57.996084 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":81,"threads_started":1}
I20260926 01:09:57.996155 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:57.991868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33981 (local address 127.0.0.1:53258)
0926 01:09:57.992009 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0926 01:09:57.992019 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 01:09:57.992104 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:09:57.992736 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:09:57.992737 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:09:57.992740 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:09:57.992790 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:09:57.992791 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:09:57.994032 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:09:57.994033 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:09:57.995067 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:09:57.995069 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:09:57.995950 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:09:57.995951 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:09:57.996007 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:09:57.996010 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 01:09:57.996049 (+ 39us) client_negotiation.cc:240] Negotiation successful
0926 01:09:57.996079 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":39,"threads_started":1}
I20260926 01:09:57.996160 1430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:57.991649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46415 (local address 127.0.0.1:43374)
0926 01:09:57.991778 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0926 01:09:57.991816 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 01:09:57.991882 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:09:57.992423 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:09:57.992424 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:09:57.992426 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:09:57.992473 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:09:57.992475 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:09:57.993614 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:09:57.993615 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:09:57.994934 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:09:57.994937 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:09:57.995893 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:09:57.995894 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:09:57.995955 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:09:57.995959 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 01:09:57.996010 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 01:09:57.996064 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":84,"threads_started":1}
I20260926 01:09:57.996388 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:57.992030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53258 (local address 127.0.98.2:33981)
0926 01:09:57.992535 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0926 01:09:57.992539 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 01:09:57.992547 (+ 8us) server_negotiation.cc:407] Connection header received
0926 01:09:57.992581 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:09:57.992583 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:09:57.992597 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:09:57.992665 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:09:57.992867 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:57.993940 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:57.995146 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:57.995823 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:57.995879 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:09:57.995923 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0926 01:09:57.996266 (+ 343us) server_negotiation.cc:299] Negotiation successful
0926 01:09:57.996311 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":84,"threads_started":1}
I20260926 01:09:57.996562 1355 tablet_service.cc:1467] Processing CreateTablet for tablet 2468d959431e411c8d067ac4c8c2013f (DEFAULT_TABLE table=test-table [id=a10689f989794c939795a2af27f2856d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:09:57.996932 1355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2468d959431e411c8d067ac4c8c2013f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:09:57.996933 1087 tablet_service.cc:1467] Processing CreateTablet for tablet 2468d959431e411c8d067ac4c8c2013f (DEFAULT_TABLE table=test-table [id=a10689f989794c939795a2af27f2856d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:09:57.996937 1221 tablet_service.cc:1467] Processing CreateTablet for tablet 2468d959431e411c8d067ac4c8c2013f (DEFAULT_TABLE table=test-table [id=a10689f989794c939795a2af27f2856d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:09:57.997270 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2468d959431e411c8d067ac4c8c2013f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:09:57.997270 1221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2468d959431e411c8d067ac4c8c2013f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:09:57.999733 1435 tablet_bootstrap.cc:492] T 2468d959431e411c8d067ac4c8c2013f P 2c688214a3c043e993f2266ece210fb2: Bootstrap starting.
I20260926 01:09:58.000560 1437 tablet_bootstrap.cc:492] T 2468d959431e411c8d067ac4c8c2013f P 47e09b1a8ead447ab88345139de3526a: Bootstrap starting.
I20260926 01:09:58.000635 1435 tablet_bootstrap.cc:654] T 2468d959431e411c8d067ac4c8c2013f P 2c688214a3c043e993f2266ece210fb2: Neither blocks nor log segments found. Creating new log.
I20260926 01:09:58.001173 1435 log.cc:824] T 2468d959431e411c8d067ac4c8c2013f P 2c688214a3c043e993f2266ece210fb2: Log is configured to *not* fsync() on all Append() calls
I20260926 01:09:58.001214 1437 tablet_bootstrap.cc:654] T 2468d959431e411c8d067ac4c8c2013f P 47e09b1a8ead447ab88345139de3526a: Neither blocks nor log segments found. Creating new log.
I20260926 01:09:58.001888 1437 log.cc:824] T 2468d959431e411c8d067ac4c8c2013f P 47e09b1a8ead447ab88345139de3526a: Log is configured to *not* fsync() on all Append() calls
I20260926 01:09:58.001960 1436 tablet_bootstrap.cc:492] T 2468d959431e411c8d067ac4c8c2013f P 4b7e8d481c9a4588bd273bad8696c41f: Bootstrap starting.
I20260926 01:09:58.002117 1435 tablet_bootstrap.cc:492] T 2468d959431e411c8d067ac4c8c2013f P 2c688214a3c043e993f2266ece210fb2: No bootstrap required, opened a new log
I20260926 01:09:58.002209 1435 ts_tablet_manager.cc:1397] T 2468d959431e411c8d067ac4c8c2013f P 2c688214a3c043e993f2266ece210fb2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 01:09:58.002589 1436 tablet_bootstrap.cc:654] T 2468d959431e411c8d067ac4c8c2013f P 4b7e8d481c9a4588bd273bad8696c41f: Neither blocks nor log segments found. Creating new log.
I20260926 01:09:58.002744 1437 tablet_bootstrap.cc:492] T 2468d959431e411c8d067ac4c8c2013f P 47e09b1a8ead447ab88345139de3526a: No bootstrap required, opened a new log
I20260926 01:09:58.002815 1437 ts_tablet_manager.cc:1397] T 2468d959431e411c8d067ac4c8c2013f P 47e09b1a8ead447ab88345139de3526a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260926 01:09:58.003091 1436 log.cc:824] T 2468d959431e411c8d067ac4c8c2013f P 4b7e8d481c9a4588bd273bad8696c41f: Log is configured to *not* fsync() on all Append() calls
I20260926 01:09:58.003808 1436 tablet_bootstrap.cc:492] T 2468d959431e411c8d067ac4c8c2013f P 4b7e8d481c9a4588bd273bad8696c41f: No bootstrap required, opened a new log
I20260926 01:09:58.003921 1436 ts_tablet_manager.cc:1397] T 2468d959431e411c8d067ac4c8c2013f P 4b7e8d481c9a4588bd273bad8696c41f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 01:09:58.004346 1435 raft_consensus.cc:348] T 2468d959431e411c8d067ac4c8c2013f P 2c688214a3c043e993f2266ece210fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b7e8d481c9a4588bd273bad8696c41f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46415 } } peers { permanent_uuid: "47e09b1a8ead447ab88345139de3526a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41149 } } peers { permanent_uuid: "2c688214a3c043e993f2266ece210fb2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33981 } }
I20260926 01:09:58.004537 1435 raft_consensus.cc:374] T 2468d959431e411c8d067ac4c8c2013f P 2c688214a3c043e993f2266ece210fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:09:58.004571 1435 raft_consensus.cc:729] T 2468d959431e411c8d067ac4c8c2013f P 2c688214a3c043e993f2266ece210fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c688214a3c043e993f2266ece210fb2, State: Initialized, Role: FOLLOWER
I20260926 01:09:58.004730 1435 consensus_queue.cc:260] T 2468d959431e411c8d067ac4c8c2013f P 2c688214a3c043e993f2266ece210fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b7e8d481c9a4588bd273bad8696c41f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46415 } } peers { permanent_uuid: "47e09b1a8ead447ab88345139de3526a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41149 } } peers { permanent_uuid: "2c688214a3c043e993f2266ece210fb2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33981 } }
I20260926 01:09:58.005021 1435 ts_tablet_manager.cc:1428] T 2468d959431e411c8d067ac4c8c2013f P 2c688214a3c043e993f2266ece210fb2: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 01:09:58.005023 1437 raft_consensus.cc:348] T 2468d959431e411c8d067ac4c8c2013f P 47e09b1a8ead447ab88345139de3526a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b7e8d481c9a4588bd273bad8696c41f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46415 } } peers { permanent_uuid: "47e09b1a8ead447ab88345139de3526a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41149 } } peers { permanent_uuid: "2c688214a3c043e993f2266ece210fb2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33981 } }
I20260926 01:09:58.005185 1437 raft_consensus.cc:374] T 2468d959431e411c8d067ac4c8c2013f P 47e09b1a8ead447ab88345139de3526a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:09:58.005188 1287 heartbeater.cc:503] Master 127.0.98.62:42625 was elected leader, sending a full tablet report...
I20260926 01:09:58.005220 1437 raft_consensus.cc:729] T 2468d959431e411c8d067ac4c8c2013f P 47e09b1a8ead447ab88345139de3526a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47e09b1a8ead447ab88345139de3526a, State: Initialized, Role: FOLLOWER
I20260926 01:09:58.005306 1437 consensus_queue.cc:260] T 2468d959431e411c8d067ac4c8c2013f P 47e09b1a8ead447ab88345139de3526a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b7e8d481c9a4588bd273bad8696c41f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46415 } } peers { permanent_uuid: "47e09b1a8ead447ab88345139de3526a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41149 } } peers { permanent_uuid: "2c688214a3c043e993f2266ece210fb2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33981 } }
I20260926 01:09:58.005527 1437 ts_tablet_manager.cc:1428] T 2468d959431e411c8d067ac4c8c2013f P 47e09b1a8ead447ab88345139de3526a: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260926 01:09:58.005549 1421 heartbeater.cc:503] Master 127.0.98.62:42625 was elected leader, sending a full tablet report...
I20260926 01:09:58.005745 1436 raft_consensus.cc:348] T 2468d959431e411c8d067ac4c8c2013f P 4b7e8d481c9a4588bd273bad8696c41f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b7e8d481c9a4588bd273bad8696c41f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46415 } } peers { permanent_uuid: "47e09b1a8ead447ab88345139de3526a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41149 } } peers { permanent_uuid: "2c688214a3c043e993f2266ece210fb2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33981 } }
I20260926 01:09:58.005936 1436 raft_consensus.cc:374] T 2468d959431e411c8d067ac4c8c2013f P 4b7e8d481c9a4588bd273bad8696c41f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:09:58.005980 1436 raft_consensus.cc:729] T 2468d959431e411c8d067ac4c8c2013f P 4b7e8d481c9a4588bd273bad8696c41f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b7e8d481c9a4588bd273bad8696c41f, State: Initialized, Role: FOLLOWER
I20260926 01:09:58.006108 1436 consensus_queue.cc:260] T 2468d959431e411c8d067ac4c8c2013f P 4b7e8d481c9a4588bd273bad8696c41f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b7e8d481c9a4588bd273bad8696c41f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46415 } } peers { permanent_uuid: "47e09b1a8ead447ab88345139de3526a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41149 } } peers { permanent_uuid: "2c688214a3c043e993f2266ece210fb2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33981 } }
I20260926 01:09:58.006387 1436 ts_tablet_manager.cc:1428] T 2468d959431e411c8d067ac4c8c2013f P 4b7e8d481c9a4588bd273bad8696c41f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 01:09:58.006392 1153 heartbeater.cc:503] Master 127.0.98.62:42625 was elected leader, sending a full tablet report...
W20260926 01:09:58.070271 1288 tablet.cc:2369] T 2468d959431e411c8d067ac4c8c2013f P 2c688214a3c043e993f2266ece210fb2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 01:09:58.092985 1442 raft_consensus.cc:482] T 2468d959431e411c8d067ac4c8c2013f P 47e09b1a8ead447ab88345139de3526a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:09:58.093129 1442 raft_consensus.cc:504] T 2468d959431e411c8d067ac4c8c2013f P 47e09b1a8ead447ab88345139de3526a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b7e8d481c9a4588bd273bad8696c41f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46415 } } peers { permanent_uuid: "47e09b1a8ead447ab88345139de3526a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41149 } } peers { permanent_uuid: "2c688214a3c043e993f2266ece210fb2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33981 } }
I20260926 01:09:58.093498 1442 leader_election.cc:288] T 2468d959431e411c8d067ac4c8c2013f P 47e09b1a8ead447ab88345139de3526a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4b7e8d481c9a4588bd273bad8696c41f (127.0.98.1:46415), 2c688214a3c043e993f2266ece210fb2 (127.0.98.2:33981)
I20260926 01:09:58.096307 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:58.093713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46415 (local address 127.0.98.3:51885)
0926 01:09:58.093763 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0926 01:09:58.093775 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 01:09:58.093842 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:09:58.094058 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:09:58.094059 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:09:58.094062 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:09:58.094136 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:09:58.094138 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:09:58.094749 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:09:58.094750 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:09:58.095504 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:09:58.095506 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:09:58.096047 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:09:58.096048 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:09:58.096141 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:09:58.096144 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 01:09:58.096191 (+ 47us) client_negotiation.cc:240] Negotiation successful
0926 01:09:58.096230 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31,"mutex_wait_us":14}
I20260926 01:09:58.096307 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:58.093791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51885 (local address 127.0.98.1:46415)
0926 01:09:58.093909 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0926 01:09:58.093912 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:09:58.093922 (+ 10us) server_negotiation.cc:407] Connection header received
0926 01:09:58.093953 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:09:58.093953 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:09:58.093965 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:09:58.094039 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:09:58.094201 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:58.094670 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:58.095597 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:58.095969 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:58.096030 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:09:58.096041 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 01:09:58.096204 (+ 163us) server_negotiation.cc:299] Negotiation successful
0926 01:09:58.096246 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260926 01:09:58.096622 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2468d959431e411c8d067ac4c8c2013f" candidate_uuid: "47e09b1a8ead447ab88345139de3526a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b7e8d481c9a4588bd273bad8696c41f" is_pre_election: true
I20260926 01:09:58.096767 1107 raft_consensus.cc:2454] T 2468d959431e411c8d067ac4c8c2013f P 4b7e8d481c9a4588bd273bad8696c41f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47e09b1a8ead447ab88345139de3526a in term 0.
I20260926 01:09:58.096959 1444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:58.093841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33981 (local address 127.0.98.3:48147)
0926 01:09:58.094025 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0926 01:09:58.094033 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 01:09:58.094087 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:09:58.094201 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:09:58.094202 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:09:58.094203 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:09:58.094270 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:09:58.094271 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:09:58.094860 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:09:58.094861 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:09:58.096068 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:09:58.096071 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:09:58.096708 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:09:58.096710 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:09:58.096780 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:09:58.096784 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 01:09:58.096856 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 01:09:58.096893 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":99,"threads_started":1}
I20260926 01:09:58.096985 1311 leader_election.cc:302] T 2468d959431e411c8d067ac4c8c2013f P 47e09b1a8ead447ab88345139de3526a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47e09b1a8ead447ab88345139de3526a, 4b7e8d481c9a4588bd273bad8696c41f; no voters:
I20260926 01:09:58.097100 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:58.093916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48147 (local address 127.0.98.2:33981)
0926 01:09:58.093970 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 01:09:58.093972 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:09:58.094095 (+ 123us) server_negotiation.cc:407] Connection header received
0926 01:09:58.094133 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:09:58.094134 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:09:58.094146 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:09:58.094197 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:09:58.094330 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:58.094789 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:58.096167 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:58.096643 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:58.096706 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:09:58.096715 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 01:09:58.096990 (+ 275us) server_negotiation.cc:299] Negotiation successful
0926 01:09:58.097039 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260926 01:09:58.097250 1442 raft_consensus.cc:2781] T 2468d959431e411c8d067ac4c8c2013f P 47e09b1a8ead447ab88345139de3526a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:09:58.097298 1442 raft_consensus.cc:482] T 2468d959431e411c8d067ac4c8c2013f P 47e09b1a8ead447ab88345139de3526a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:09:58.097319 1442 raft_consensus.cc:3037] T 2468d959431e411c8d067ac4c8c2013f P 47e09b1a8ead447ab88345139de3526a [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:09:58.097352 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2468d959431e411c8d067ac4c8c2013f" candidate_uuid: "47e09b1a8ead447ab88345139de3526a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c688214a3c043e993f2266ece210fb2" is_pre_election: true
I20260926 01:09:58.097496 1241 raft_consensus.cc:2454] T 2468d959431e411c8d067ac4c8c2013f P 2c688214a3c043e993f2266ece210fb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47e09b1a8ead447ab88345139de3526a in term 0.
I20260926 01:09:58.098098 1442 raft_consensus.cc:504] T 2468d959431e411c8d067ac4c8c2013f P 47e09b1a8ead447ab88345139de3526a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b7e8d481c9a4588bd273bad8696c41f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46415 } } peers { permanent_uuid: "47e09b1a8ead447ab88345139de3526a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41149 } } peers { permanent_uuid: "2c688214a3c043e993f2266ece210fb2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33981 } }
I20260926 01:09:58.098212 1442 leader_election.cc:288] T 2468d959431e411c8d067ac4c8c2013f P 47e09b1a8ead447ab88345139de3526a [CANDIDATE]: Term 1 election: Requested vote from peers 4b7e8d481c9a4588bd273bad8696c41f (127.0.98.1:46415), 2c688214a3c043e993f2266ece210fb2 (127.0.98.2:33981)
I20260926 01:09:58.098343 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2468d959431e411c8d067ac4c8c2013f" candidate_uuid: "47e09b1a8ead447ab88345139de3526a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b7e8d481c9a4588bd273bad8696c41f"
I20260926 01:09:58.098381 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2468d959431e411c8d067ac4c8c2013f" candidate_uuid: "47e09b1a8ead447ab88345139de3526a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c688214a3c043e993f2266ece210fb2"
I20260926 01:09:58.098414 1107 raft_consensus.cc:3037] T 2468d959431e411c8d067ac4c8c2013f P 4b7e8d481c9a4588bd273bad8696c41f [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:09:58.098451 1241 raft_consensus.cc:3037] T 2468d959431e411c8d067ac4c8c2013f P 2c688214a3c043e993f2266ece210fb2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:09:58.099120 1107 raft_consensus.cc:2454] T 2468d959431e411c8d067ac4c8c2013f P 4b7e8d481c9a4588bd273bad8696c41f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47e09b1a8ead447ab88345139de3526a in term 1.
I20260926 01:09:58.099175 1241 raft_consensus.cc:2454] T 2468d959431e411c8d067ac4c8c2013f P 2c688214a3c043e993f2266ece210fb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47e09b1a8ead447ab88345139de3526a in term 1.
I20260926 01:09:58.099298 1311 leader_election.cc:302] T 2468d959431e411c8d067ac4c8c2013f P 47e09b1a8ead447ab88345139de3526a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c688214a3c043e993f2266ece210fb2, 47e09b1a8ead447ab88345139de3526a; no voters:
I20260926 01:09:58.099387 1442 raft_consensus.cc:2781] T 2468d959431e411c8d067ac4c8c2013f P 47e09b1a8ead447ab88345139de3526a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:09:58.099560 1442 raft_consensus.cc:686] T 2468d959431e411c8d067ac4c8c2013f P 47e09b1a8ead447ab88345139de3526a [term 1 LEADER]: Becoming Leader. State: Replica: 47e09b1a8ead447ab88345139de3526a, State: Running, Role: LEADER
I20260926 01:09:58.099678 1442 consensus_queue.cc:237] T 2468d959431e411c8d067ac4c8c2013f P 47e09b1a8ead447ab88345139de3526a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b7e8d481c9a4588bd273bad8696c41f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46415 } } peers { permanent_uuid: "47e09b1a8ead447ab88345139de3526a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41149 } } peers { permanent_uuid: "2c688214a3c043e993f2266ece210fb2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33981 } }
I20260926 01:09:58.100479 961 catalog_manager.cc:5495] T 2468d959431e411c8d067ac4c8c2013f P 47e09b1a8ead447ab88345139de3526a reported cstate change: term changed from 0 to 1, leader changed from <none> to 47e09b1a8ead447ab88345139de3526a (127.0.98.3). New cstate: current_term: 1 leader_uuid: "47e09b1a8ead447ab88345139de3526a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b7e8d481c9a4588bd273bad8696c41f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47e09b1a8ead447ab88345139de3526a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41149 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2c688214a3c043e993f2266ece210fb2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33981 } health_report { overall_health: UNKNOWN } } }
Sep 26 01:09:58 dist-test-slave-7b22 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384997, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 01:09:58.116037 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:58.108846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43376 (local address 127.0.98.1:46415)
0926 01:09:58.108895 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 01:09:58.108897 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:09:58.108993 (+ 96us) server_negotiation.cc:407] Connection header received
0926 01:09:58.109059 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:09:58.109060 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:09:58.109071 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:09:58.109122 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:09:58.109624 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:58.110245 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:58.110930 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:58.111109 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:58.111176 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:09:58.113399 (+ 2223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:09:58.113409 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:09:58.113410 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:09:58.113417 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:09:58.115005 (+ 1588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:09:58.115363 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:09:58.115366 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:09:58.115367 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:09:58.115425 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:09:58.115598 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:09:58.115598 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:09:58.115599 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:09:58.115733 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:09:58.115832 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 01:09:58.115887 (+ 55us) server_negotiation.cc:299] Negotiation successful
0926 01:09:58.115948 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
Sep 26 01:09:58 dist-test-slave-7b22 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384998, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 01:09:58 dist-test-slave-7b22 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384998, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 01:09:58.137740 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:58.130763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43388 (local address 127.0.98.1:46415)
0926 01:09:58.130843 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0926 01:09:58.130845 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:09:58.130892 (+ 47us) server_negotiation.cc:407] Connection header received
0926 01:09:58.130947 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:09:58.130948 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:09:58.130957 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:09:58.131005 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:09:58.131546 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:58.132068 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:58.132835 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:58.133042 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:58.133116 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:09:58.135337 (+ 2221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:09:58.135339 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:09:58.135339 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:09:58.135346 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:09:58.136670 (+ 1324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:09:58.136978 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:09:58.136979 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:09:58.136980 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:09:58.137033 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:09:58.137249 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:09:58.137250 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:09:58.137250 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:09:58.137352 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:09:58.137431 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 01:09:58.137588 (+ 157us) server_negotiation.cc:299] Negotiation successful
0926 01:09:58.137650 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
Sep 26 01:09:58 dist-test-slave-7b22 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384998, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 01:09:58 dist-test-slave-7b22 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384998, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 01:09:58.160687 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:58.154114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43396 (local address 127.0.98.1:46415)
0926 01:09:58.154177 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 01:09:58.154179 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:09:58.154246 (+ 67us) server_negotiation.cc:407] Connection header received
0926 01:09:58.154326 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:09:58.154327 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:09:58.154337 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:09:58.154383 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:09:58.154944 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:58.155426 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:58.156139 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:58.156332 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:58.156412 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:09:58.158588 (+ 2176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:09:58.158590 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:09:58.158590 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:09:58.158596 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:09:58.159799 (+ 1203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:09:58.160124 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:09:58.160125 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:09:58.160126 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:09:58.160184 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:09:58.160322 (+ 138us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:09:58.160323 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:09:58.160323 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:09:58.160411 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:09:58.160481 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 01:09:58.160556 (+ 75us) server_negotiation.cc:299] Negotiation successful
0926 01:09:58.160619 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
W20260926 01:09:58.161114 1067 tablet_service.cc:2908] Rejecting scan request for tablet 2468d959431e411c8d067ac4c8c2013f: Uninitialized: clean time has not yet been initialized
I20260926 01:09:58.161990 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 1024
I20260926 01:09:58.167153 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 1158
I20260926 01:09:58.171120 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 1292
I20260926 01:09:58.175168 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 930
2026-09-26T01:09:58Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (1873 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384993574762-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 01:09:58 dist-test-slave-7b22 krb5kdc[1459](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 01:09:58 dist-test-slave-7b22 krb5kdc[1459](info): set up 2 sockets
Sep 26 01:09:58 dist-test-slave-7b22 krb5kdc[1459](info): commencing operation
krb5kdc: starting...
W20260926 01:09:59.129245 392 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384993574762-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384993574762-392-0/krb5kdc/test-user.keytab.
Sep 26 01:09:59 dist-test-slave-7b22 krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384999, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T01:09:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:09:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384993574762-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384993574762-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 01:09:59.242975 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384993574762-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384993574762-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384993574762-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42075
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384993574762-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42075
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:09:59.323264 1472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:09:59.323532 1472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:09:59.323570 1472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:09:59.324764 1472 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:09:59.324805 1472 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 01:09:59.324821 1472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:09:59.324838 1472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:09:59.324854 1472 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:09:59.324870 1472 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:09:59.326520 1472 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384993574762-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384993574762-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42075
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384993574762-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42075
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1472
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384993574762-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:09:59.326759 1472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:09:59.326996 1472 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:09:59.331974 1480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:09:59.331967 1477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:09:59.331962 1478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:09:59.331998 1472 server_base.cc:1034] running on GCE node
I20260926 01:09:59.332408 1472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:09:59.332692 1472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:09:59.333866 1472 hybrid_clock.cc:648] HybridClock initialized: now 1790384999333845 us; error 33 us; skew 500 ppm
Sep 26 01:09:59 dist-test-slave-7b22 krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384999, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:09:59.336297 1472 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 01:09:59.336740 1472 webserver.cc:460] Webserver started at http://127.0.98.62:42713/ using document root <none> and password file <none>
I20260926 01:09:59.336956 1472 fs_manager.cc:359] Metadata directory not provided
I20260926 01:09:59.337002 1472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:09:59.337143 1472 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:09:59.338260 1472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384993574762-392-0/minicluster-data/master-0/data/instance:
uuid: "a3fc3ff0f3ad4ec19518afe126ee308f"
format_stamp: "Formatted at 2026-09-26 01:09:59 on dist-test-slave-7b22"
server_key: "bc7c504e8de927cfc1be16dad68d1dfb"
server_key_iv: "7a72e2f44c5a87c23ef5b34f1cee7cf6"
server_key_version: "encryptionkey@0"
I20260926 01:09:59.338668 1472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384993574762-392-0/minicluster-data/master-0/wal/instance:
uuid: "a3fc3ff0f3ad4ec19518afe126ee308f"
format_stamp: "Formatted at 2026-09-26 01:09:59 on dist-test-slave-7b22"
server_key: "bc7c504e8de927cfc1be16dad68d1dfb"
server_key_iv: "7a72e2f44c5a87c23ef5b34f1cee7cf6"
server_key_version: "encryptionkey@0"
I20260926 01:09:59.339900 1472 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 01:09:59.340692 1487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:09:59.340855 1472 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:09:59.340939 1472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384993574762-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384993574762-392-0/minicluster-data/master-0/wal
uuid: "a3fc3ff0f3ad4ec19518afe126ee308f"
format_stamp: "Formatted at 2026-09-26 01:09:59 on dist-test-slave-7b22"
server_key: "bc7c504e8de927cfc1be16dad68d1dfb"
server_key_iv: "7a72e2f44c5a87c23ef5b34f1cee7cf6"
server_key_version: "encryptionkey@0"
I20260926 01:09:59.340997 1472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384993574762-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384993574762-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384993574762-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:09:59.383103 1472 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:09:59.385149 1472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:09:59.385316 1472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:09:59.389763 1472 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42075
I20260926 01:09:59.389837 1549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42075 every 8 connection(s)
I20260926 01:09:59.390170 1472 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
I20260926 01:09:59.390893 1550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:09:59.393213 1550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3fc3ff0f3ad4ec19518afe126ee308f: Bootstrap starting.
I20260926 01:09:59.393800 1550 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a3fc3ff0f3ad4ec19518afe126ee308f: Neither blocks nor log segments found. Creating new log.
I20260926 01:09:59.394121 1550 log.cc:824] T 00000000000000000000000000000000 P a3fc3ff0f3ad4ec19518afe126ee308f: Log is configured to *not* fsync() on all Append() calls
I20260926 01:09:59.394837 1550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3fc3ff0f3ad4ec19518afe126ee308f: No bootstrap required, opened a new log
I20260926 01:09:59.396852 1550 raft_consensus.cc:348] T 00000000000000000000000000000000 P a3fc3ff0f3ad4ec19518afe126ee308f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3fc3ff0f3ad4ec19518afe126ee308f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42075 } }
I20260926 01:09:59.396996 1550 raft_consensus.cc:374] T 00000000000000000000000000000000 P a3fc3ff0f3ad4ec19518afe126ee308f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:09:59.397019 1550 raft_consensus.cc:729] T 00000000000000000000000000000000 P a3fc3ff0f3ad4ec19518afe126ee308f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3fc3ff0f3ad4ec19518afe126ee308f, State: Initialized, Role: FOLLOWER
I20260926 01:09:59.397159 1550 consensus_queue.cc:260] T 00000000000000000000000000000000 P a3fc3ff0f3ad4ec19518afe126ee308f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3fc3ff0f3ad4ec19518afe126ee308f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42075 } }
I20260926 01:09:59.397228 1550 raft_consensus.cc:388] T 00000000000000000000000000000000 P a3fc3ff0f3ad4ec19518afe126ee308f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:09:59.397248 1550 raft_consensus.cc:482] T 00000000000000000000000000000000 P a3fc3ff0f3ad4ec19518afe126ee308f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:09:59.397292 1550 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a3fc3ff0f3ad4ec19518afe126ee308f [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:09:59.397851 1550 raft_consensus.cc:504] T 00000000000000000000000000000000 P a3fc3ff0f3ad4ec19518afe126ee308f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3fc3ff0f3ad4ec19518afe126ee308f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42075 } }
I20260926 01:09:59.397954 1550 leader_election.cc:302] T 00000000000000000000000000000000 P a3fc3ff0f3ad4ec19518afe126ee308f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a3fc3ff0f3ad4ec19518afe126ee308f; no voters:
I20260926 01:09:59.398051 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 1472
I20260926 01:09:59.398116 1550 leader_election.cc:288] T 00000000000000000000000000000000 P a3fc3ff0f3ad4ec19518afe126ee308f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:09:59.398141 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384993574762-392-0/minicluster-data/master-0/wal/instance
I20260926 01:09:59.398164 1553 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a3fc3ff0f3ad4ec19518afe126ee308f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:09:59.398293 392 external_mini_cluster.cc:1428] Setting key 96567a64a7c30de5eb943cf0fca737d1
I20260926 01:09:59.398293 1553 raft_consensus.cc:686] T 00000000000000000000000000000000 P a3fc3ff0f3ad4ec19518afe126ee308f [term 1 LEADER]: Becoming Leader. State: Replica: a3fc3ff0f3ad4ec19518afe126ee308f, State: Running, Role: LEADER
I20260926 01:09:59.398371 1553 consensus_queue.cc:237] T 00000000000000000000000000000000 P a3fc3ff0f3ad4ec19518afe126ee308f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3fc3ff0f3ad4ec19518afe126ee308f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42075 } }
I20260926 01:09:59.398435 1550 sys_catalog.cc:564] T 00000000000000000000000000000000 P a3fc3ff0f3ad4ec19518afe126ee308f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:09:59.398669 1553 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3fc3ff0f3ad4ec19518afe126ee308f [sys.catalog]: SysCatalogTable state changed. Reason: New leader a3fc3ff0f3ad4ec19518afe126ee308f. Latest consensus state: current_term: 1 leader_uuid: "a3fc3ff0f3ad4ec19518afe126ee308f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3fc3ff0f3ad4ec19518afe126ee308f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42075 } } }
I20260926 01:09:59.398828 1553 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3fc3ff0f3ad4ec19518afe126ee308f [sys.catalog]: This master's current role is: LEADER
I20260926 01:09:59.398702 1554 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3fc3ff0f3ad4ec19518afe126ee308f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a3fc3ff0f3ad4ec19518afe126ee308f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3fc3ff0f3ad4ec19518afe126ee308f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42075 } } }
I20260926 01:09:59.399072 1554 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3fc3ff0f3ad4ec19518afe126ee308f [sys.catalog]: This master's current role is: LEADER
I20260926 01:09:59.399057 1561 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:09:59.399700 1561 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:09:59.401302 1561 catalog_manager.cc:1347] Generated new cluster ID: 3f4b4365e7da4a9eb7b145c061594139
I20260926 01:09:59.401346 1561 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 01:09:59 dist-test-slave-7b22 krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384999, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:09:59.410665 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:59.398856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38130 (local address 127.0.98.62:42075)
0926 01:09:59.399786 (+ 930us) server_negotiation.cc:206] Beginning negotiation
0926 01:09:59.399796 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 01:09:59.399836 (+ 40us) server_negotiation.cc:407] Connection header received
0926 01:09:59.399969 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:09:59.399977 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:09:59.400040 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:09:59.400190 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:09:59.400825 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:59.402215 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:59.403239 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:59.403895 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:59.404010 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:09:59.406602 (+ 2592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:09:59.406607 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:09:59.406608 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:09:59.406622 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:09:59.408709 (+ 2087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:09:59.409165 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:09:59.409168 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:09:59.409169 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:09:59.409290 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:09:59.409472 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:09:59.409473 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:09:59.409474 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:09:59.409803 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:09:59.410000 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0926 01:09:59.410267 (+ 267us) server_negotiation.cc:299] Negotiation successful
0926 01:09:59.410415 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":873,"thread_start_us":73,"threads_started":1}
I20260926 01:09:59.414785 1561 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:09:59.415489 1561 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:09:59.420629 1561 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a3fc3ff0f3ad4ec19518afe126ee308f: Generated new TSK 0
I20260926 01:09:59.420832 1561 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384993574762-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384993574762-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 01:09:59.449496 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384993574762-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384993574762-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384993574762-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384993574762-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42075
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384993574762-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:09:59.534139 1576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:09:59.534346 1576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:09:59.534371 1576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:09:59.535598 1576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:09:59.535645 1576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:09:59.535665 1576 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 01:09:59.535681 1576 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 01:09:59.537276 1576 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384993574762-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384993574762-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384993574762-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384993574762-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:42075
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1576
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384993574762-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:09:59.537533 1576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:09:59.537762 1576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:09:59.542533 1581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:09:59.542510 1582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:09:59.542515 1584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:09:59.542789 1576 server_base.cc:1034] running on GCE node
I20260926 01:09:59.543059 1576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:09:59.543295 1576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:09:59.544464 1576 hybrid_clock.cc:648] HybridClock initialized: now 1790384999544448 us; error 33 us; skew 500 ppm
Sep 26 01:09:59 dist-test-slave-7b22 krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384999, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:09:59.546981 1576 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 01:09:59.547439 1576 webserver.cc:460] Webserver started at http://127.0.98.1:42871/ using document root <none> and password file <none>
I20260926 01:09:59.547703 1576 fs_manager.cc:359] Metadata directory not provided
I20260926 01:09:59.547755 1576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:09:59.547883 1576 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:09:59.548951 1576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384993574762-392-0/minicluster-data/ts-0/data/instance:
uuid: "04f5f856ae004e06bfe8d74881ec7371"
format_stamp: "Formatted at 2026-09-26 01:09:59 on dist-test-slave-7b22"
server_key: "219f972d02654f22959a93b53ef78390"
server_key_iv: "2c6f7b7a357cd1e1022908a7dd90acfb"
server_key_version: "encryptionkey@0"
I20260926 01:09:59.549293 1576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384993574762-392-0/minicluster-data/ts-0/wal/instance:
uuid: "04f5f856ae004e06bfe8d74881ec7371"
format_stamp: "Formatted at 2026-09-26 01:09:59 on dist-test-slave-7b22"
server_key: "219f972d02654f22959a93b53ef78390"
server_key_iv: "2c6f7b7a357cd1e1022908a7dd90acfb"
server_key_version: "encryptionkey@0"
I20260926 01:09:59.550633 1576 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 01:09:59.551355 1591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:09:59.551499 1576 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 01:09:59.551632 1576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384993574762-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384993574762-392-0/minicluster-data/ts-0/wal
uuid: "04f5f856ae004e06bfe8d74881ec7371"
format_stamp: "Formatted at 2026-09-26 01:09:59 on dist-test-slave-7b22"
server_key: "219f972d02654f22959a93b53ef78390"
server_key_iv: "2c6f7b7a357cd1e1022908a7dd90acfb"
server_key_version: "encryptionkey@0"
I20260926 01:09:59.551695 1576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384993574762-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384993574762-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384993574762-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:09:59.570818 1576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:09:59.573232 1576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:09:59.573387 1576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:09:59.573971 1576 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:09:59.574023 1576 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:09:59.574059 1576 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:09:59.574085 1576 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 01:09:59 dist-test-slave-7b22 krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384999, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:09:59.581314 1576 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42393
I20260926 01:09:59.581765 1576 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384993574762-392-0/minicluster-data/ts-0/data/info.pb
I20260926 01:09:59.581765 1706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42393 every 8 connection(s)
I20260926 01:09:59.583894 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:59.574421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57883 (local address 127.0.98.62:42075)
0926 01:09:59.574495 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0926 01:09:59.574498 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:09:59.574693 (+ 195us) server_negotiation.cc:407] Connection header received
0926 01:09:59.575220 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:09:59.575222 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:09:59.575235 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:09:59.575288 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:09:59.575972 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:59.576485 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:59.577483 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:59.577705 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:59.577760 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:09:59.580881 (+ 3121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:09:59.580884 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:09:59.580884 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:09:59.580894 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:09:59.582541 (+ 1647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:09:59.582974 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:09:59.582975 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:09:59.582976 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:09:59.583036 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:09:59.583322 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:09:59.583323 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:09:59.583324 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:09:59.583451 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:09:59.583520 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 01:09:59.583746 (+ 226us) server_negotiation.cc:299] Negotiation successful
0926 01:09:59.583829 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260926 01:09:59.584044 1600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:59.574428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42075 (local address 127.0.98.1:57883)
0926 01:09:59.574577 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0926 01:09:59.574614 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 01:09:59.575118 (+ 504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:09:59.575367 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:09:59.575371 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:09:59.575647 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:09:59.575879 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:09:59.575884 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:09:59.576573 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:09:59.576574 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:09:59.577398 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:09:59.577400 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:09:59.577777 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:09:59.577778 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:09:59.577953 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:09:59.578470 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:09:59.578480 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:09:59.580773 (+ 2293us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:09:59.582613 (+ 1840us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:09:59.582617 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:09:59.582618 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:09:59.582900 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:09:59.583117 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:09:59.583118 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:09:59.583118 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:09:59.583231 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:09:59.583513 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:09:59.583515 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:09:59.583644 (+ 129us) client_negotiation.cc:769] Sending connection context
0926 01:09:59.583721 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 01:09:59.583851 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":72,"threads_started":1}
I20260926 01:09:59.584569 1707 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42075
I20260926 01:09:59.584659 1707 heartbeater.cc:464] Registering TS with master...
I20260926 01:09:59.584779 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 1576
I20260926 01:09:59.584877 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384993574762-392-0/minicluster-data/ts-0/wal/instance
I20260926 01:09:59.584900 1707 heartbeater.cc:511] Master 127.0.98.62:42075 requested a full tablet report, sending...
I20260926 01:09:59.585069 392 external_mini_cluster.cc:1428] Setting key 0bb5bd07284f6508bfb0b99f14dda9ba
I20260926 01:09:59.585492 1503 ts_manager.cc:194] Registered new tserver with Master: 04f5f856ae004e06bfe8d74881ec7371 (127.0.98.1:42393)
I20260926 01:09:59.586247 1503 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:57883
I20260926 01:09:59.591174 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:59.585592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56391 (local address 127.0.98.62:42075)
0926 01:09:59.585747 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0926 01:09:59.585749 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:09:59.585755 (+ 6us) server_negotiation.cc:407] Connection header received
0926 01:09:59.585772 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:09:59.585773 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:09:59.585782 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:09:59.585857 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:09:59.586183 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:59.586859 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:59.587617 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:59.587891 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:59.587945 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:09:59.588894 (+ 949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:09:59.588895 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:09:59.588896 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:09:59.588902 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:09:59.590169 (+ 1267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:09:59.590520 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:09:59.590520 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:09:59.590520 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:09:59.590569 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:09:59.590754 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:09:59.590754 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:09:59.590755 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:09:59.590847 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:09:59.590890 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0926 01:09:59.591056 (+ 166us) server_negotiation.cc:299] Negotiation successful
0926 01:09:59.591112 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136}
I20260926 01:09:59.591243 1713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:59.585535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42075 (local address 127.0.98.1:56391)
0926 01:09:59.585662 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0926 01:09:59.585671 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 01:09:59.585711 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:09:59.585872 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:09:59.585873 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:09:59.586052 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:09:59.586115 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:09:59.586118 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:09:59.586965 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:09:59.586966 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:09:59.587537 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:09:59.587539 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:09:59.587961 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:09:59.587962 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:09:59.588057 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:09:59.588448 (+ 391us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:09:59.588452 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:09:59.588817 (+ 365us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:09:59.590241 (+ 1424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:09:59.590242 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:09:59.590243 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:09:59.590452 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:09:59.590627 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:09:59.590628 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:09:59.590629 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:09:59.590701 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:09:59.590896 (+ 195us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:09:59.590897 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:09:59.590946 (+ 49us) client_negotiation.cc:769] Sending connection context
0926 01:09:59.591031 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 01:09:59.591134 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":71,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384993574762-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384993574762-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 01:09:59.610745 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384993574762-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384993574762-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384993574762-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384993574762-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42075
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384993574762-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:09:59.692030 1716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:09:59.692253 1716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:09:59.692278 1716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:09:59.693521 1716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:09:59.693569 1716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:09:59.693590 1716 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 01:09:59.693607 1716 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 01:09:59.695302 1716 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384993574762-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384993574762-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384993574762-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384993574762-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:42075
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1716
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384993574762-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:09:59.695538 1716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:09:59.695791 1716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:09:59.700958 1724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:09:59.700944 1721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:09:59.701150 1716 server_base.cc:1034] running on GCE node
W20260926 01:09:59.700944 1722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:09:59.701393 1716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:09:59.701620 1716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:09:59.702798 1716 hybrid_clock.cc:648] HybridClock initialized: now 1790384999702612 us; error 200 us; skew 500 ppm
Sep 26 01:09:59 dist-test-slave-7b22 krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384999, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:09:59.705467 1716 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 01:09:59.705982 1716 webserver.cc:460] Webserver started at http://127.0.98.2:37273/ using document root <none> and password file <none>
I20260926 01:09:59.706231 1716 fs_manager.cc:359] Metadata directory not provided
I20260926 01:09:59.706284 1716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:09:59.706406 1716 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:09:59.707476 1716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384993574762-392-0/minicluster-data/ts-1/data/instance:
uuid: "f873f7c4a5944baab02c621fee691375"
format_stamp: "Formatted at 2026-09-26 01:09:59 on dist-test-slave-7b22"
server_key: "a18ab0fa3d49ee019aea6d2f856078cb"
server_key_iv: "a7ebcca42b56b665cba52a32b768baf7"
server_key_version: "encryptionkey@0"
I20260926 01:09:59.707806 1716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384993574762-392-0/minicluster-data/ts-1/wal/instance:
uuid: "f873f7c4a5944baab02c621fee691375"
format_stamp: "Formatted at 2026-09-26 01:09:59 on dist-test-slave-7b22"
server_key: "a18ab0fa3d49ee019aea6d2f856078cb"
server_key_iv: "a7ebcca42b56b665cba52a32b768baf7"
server_key_version: "encryptionkey@0"
I20260926 01:09:59.709159 1716 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 01:09:59.709926 1731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:09:59.710059 1716 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 01:09:59.710143 1716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384993574762-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384993574762-392-0/minicluster-data/ts-1/wal
uuid: "f873f7c4a5944baab02c621fee691375"
format_stamp: "Formatted at 2026-09-26 01:09:59 on dist-test-slave-7b22"
server_key: "a18ab0fa3d49ee019aea6d2f856078cb"
server_key_iv: "a7ebcca42b56b665cba52a32b768baf7"
server_key_version: "encryptionkey@0"
I20260926 01:09:59.710227 1716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384993574762-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384993574762-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384993574762-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:09:59.723289 1716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:09:59.726069 1716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:09:59.726243 1716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:09:59.726833 1716 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:09:59.726877 1716 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:09:59.726908 1716 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:09:59.726925 1716 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 01:09:59 dist-test-slave-7b22 krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384999, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:09:59.734062 1716 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37623
I20260926 01:09:59.734133 1846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37623 every 8 connection(s)
I20260926 01:09:59.734467 1716 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384993574762-392-0/minicluster-data/ts-1/data/info.pb
I20260926 01:09:59.735281 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 1716
I20260926 01:09:59.735383 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384993574762-392-0/minicluster-data/ts-1/wal/instance
I20260926 01:09:59.735569 392 external_mini_cluster.cc:1428] Setting key 8ba09ad01763c42bb0c04705af4a52e1
I20260926 01:09:59.737164 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:59.727457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42971 (local address 127.0.98.62:42075)
0926 01:09:59.727507 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 01:09:59.727509 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:09:59.727808 (+ 299us) server_negotiation.cc:407] Connection header received
0926 01:09:59.728619 (+ 811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:09:59.728620 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:09:59.728633 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:09:59.728684 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:09:59.729617 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:59.730218 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:59.731304 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:59.731518 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:59.731578 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:09:59.733983 (+ 2405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:09:59.733985 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:09:59.733985 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:09:59.733994 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:09:59.735526 (+ 1532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:09:59.736134 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:09:59.736135 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:09:59.736136 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:09:59.736209 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:09:59.736493 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:09:59.736494 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:09:59.736495 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:09:59.736637 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:09:59.736703 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 01:09:59.736968 (+ 265us) server_negotiation.cc:299] Negotiation successful
0926 01:09:59.737075 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 01:09:59.737238 1740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:59.727455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42075 (local address 127.0.98.2:42971)
0926 01:09:59.727651 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0926 01:09:59.727700 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 01:09:59.728500 (+ 800us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:09:59.728812 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:09:59.728817 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:09:59.729192 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:09:59.729504 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:09:59.729511 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:09:59.730318 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:09:59.730319 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:09:59.731211 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:09:59.731214 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:09:59.731615 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:09:59.731616 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:09:59.731748 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:09:59.732262 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:09:59.732271 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:09:59.733887 (+ 1616us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:09:59.735669 (+ 1782us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:09:59.735676 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:09:59.735678 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:09:59.736002 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:09:59.736285 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:09:59.736286 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:09:59.736287 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:09:59.736429 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:09:59.736695 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:09:59.736720 (+ 25us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:09:59.736848 (+ 128us) client_negotiation.cc:769] Sending connection context
0926 01:09:59.736932 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 01:09:59.737060 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":76,"threads_started":1}
I20260926 01:09:59.737689 1847 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42075
I20260926 01:09:59.737871 1847 heartbeater.cc:464] Registering TS with master...
I20260926 01:09:59.738071 1847 heartbeater.cc:511] Master 127.0.98.62:42075 requested a full tablet report, sending...
I20260926 01:09:59.738422 1503 ts_manager.cc:194] Registered new tserver with Master: f873f7c4a5944baab02c621fee691375 (127.0.98.2:37623)
I20260926 01:09:59.738883 1503 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:42971
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260926 01:09:59.745232 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:59.738779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40539 (local address 127.0.98.62:42075)
0926 01:09:59.738840 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 01:09:59.738842 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:09:59.739015 (+ 173us) server_negotiation.cc:407] Connection header received
0926 01:09:59.739100 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:09:59.739101 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:09:59.739111 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:09:59.739138 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:09:59.739503 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:59.740014 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:59.740781 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:59.741005 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:59.741058 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:09:59.742172 (+ 1114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:09:59.742174 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:09:59.742175 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:09:59.742182 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:09:59.744104 (+ 1922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:09:59.744460 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:09:59.744461 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:09:59.744461 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:09:59.744519 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:09:59.744703 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:09:59.744704 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:09:59.744704 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:09:59.744851 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:09:59.744910 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 01:09:59.745052 (+ 142us) server_negotiation.cc:299] Negotiation successful
0926 01:09:59.745133 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 01:09:59.745294 1854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:59.738825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42075 (local address 127.0.98.2:40539)
0926 01:09:59.738954 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0926 01:09:59.738964 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 01:09:59.739027 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:09:59.739194 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:09:59.739195 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:09:59.739374 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:09:59.739433 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:09:59.739434 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:09:59.740103 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:09:59.740104 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:09:59.740695 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:09:59.740697 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:09:59.741065 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:09:59.741066 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:09:59.741129 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:09:59.741695 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:09:59.741700 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:09:59.742081 (+ 381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:09:59.744185 (+ 2104us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:09:59.744186 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:09:59.744186 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:09:59.744398 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:09:59.744591 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:09:59.744592 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:09:59.744593 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:09:59.744651 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:09:59.744908 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:09:59.744909 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:09:59.744956 (+ 47us) client_negotiation.cc:769] Sending connection context
0926 01:09:59.745024 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 01:09:59.745135 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":71,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384993574762-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384993574762-392-0/krb5kdc/kudu_127.0.98.3.keytab.
W20260926 01:10:00.141836 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.3: real 0.405s user 0.000s sys 0.003s
I20260926 01:10:00.142149 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384993574762-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384993574762-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384993574762-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384993574762-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42075
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384993574762-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:10:00.222540 1856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:10:00.222774 1856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:10:00.222807 1856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:10:00.224032 1856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:10:00.224081 1856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:10:00.224109 1856 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 01:10:00.224133 1856 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 01:10:00.225752 1856 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384993574762-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384993574762-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384993574762-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384993574762-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:42075
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1856
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384993574762-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:10:00.226032 1856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:10:00.226244 1856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:10:00.231142 1861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:10:00.231163 1862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:10:00.231175 1864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:10:00.231487 1856 server_base.cc:1034] running on GCE node
I20260926 01:10:00.231745 1856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:10:00.231990 1856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:10:00.233150 1856 hybrid_clock.cc:648] HybridClock initialized: now 1790385000233136 us; error 30 us; skew 500 ppm
Sep 26 01:10:00 dist-test-slave-7b22 krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385000, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:10:00.235699 1856 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 01:10:00.236140 1856 webserver.cc:460] Webserver started at http://127.0.98.3:46829/ using document root <none> and password file <none>
I20260926 01:10:00.236356 1856 fs_manager.cc:359] Metadata directory not provided
I20260926 01:10:00.236404 1856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:10:00.236541 1856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:10:00.237691 1856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384993574762-392-0/minicluster-data/ts-2/data/instance:
uuid: "caf1cf8c0a49436fb92743a2cddc285c"
format_stamp: "Formatted at 2026-09-26 01:10:00 on dist-test-slave-7b22"
server_key: "80a744611023bc169e1bb4662ca35668"
server_key_iv: "cdc344f5166818ef2fd40f6fcdf6251d"
server_key_version: "encryptionkey@0"
I20260926 01:10:00.238057 1856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384993574762-392-0/minicluster-data/ts-2/wal/instance:
uuid: "caf1cf8c0a49436fb92743a2cddc285c"
format_stamp: "Formatted at 2026-09-26 01:10:00 on dist-test-slave-7b22"
server_key: "80a744611023bc169e1bb4662ca35668"
server_key_iv: "cdc344f5166818ef2fd40f6fcdf6251d"
server_key_version: "encryptionkey@0"
I20260926 01:10:00.239434 1856 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:10:00.240266 1871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:00.240454 1856 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:10:00.240542 1856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384993574762-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384993574762-392-0/minicluster-data/ts-2/wal
uuid: "caf1cf8c0a49436fb92743a2cddc285c"
format_stamp: "Formatted at 2026-09-26 01:10:00 on dist-test-slave-7b22"
server_key: "80a744611023bc169e1bb4662ca35668"
server_key_iv: "cdc344f5166818ef2fd40f6fcdf6251d"
server_key_version: "encryptionkey@0"
I20260926 01:10:00.240605 1856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384993574762-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384993574762-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384993574762-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:10:00.257624 1856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:10:00.259814 1856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:10:00.259956 1856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:10:00.260468 1856 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:10:00.260505 1856 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:00.260535 1856 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:10:00.260551 1856 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 01:10:00 dist-test-slave-7b22 krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385000, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:10:00.267202 1856 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42925
I20260926 01:10:00.267477 1987 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42925 every 8 connection(s)
I20260926 01:10:00.267622 1856 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384993574762-392-0/minicluster-data/ts-2/data/info.pb
I20260926 01:10:00.269965 1880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:00.261079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57095 (local address 127.0.98.62:42075)
0926 01:10:00.261223 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:00.261225 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:00.261367 (+ 142us) server_negotiation.cc:407] Connection header received
0926 01:10:00.262127 (+ 760us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:00.262129 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:00.262145 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:00.262236 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:00.262855 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:00.263428 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:00.264477 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:00.264683 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:00.264736 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:00.266899 (+ 2163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:10:00.266901 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:10:00.266901 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:10:00.266908 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:10:00.268467 (+ 1559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:00.268863 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:00.268863 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:00.268864 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:00.268913 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:00.269240 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:00.269242 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:00.269242 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:00.269372 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:10:00.269433 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:00.269747 (+ 314us) server_negotiation.cc:299] Negotiation successful
0926 01:10:00.269864 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":67,"threads_started":1}
I20260926 01:10:00.270226 1881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:00.261044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42075 (local address 127.0.98.3:57095)
0926 01:10:00.261237 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:00.261279 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:00.261988 (+ 709us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:00.262315 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:00.262320 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:00.262568 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:10:00.262766 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:00.262771 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:00.263514 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:00.263515 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:00.264394 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:00.264396 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:00.264757 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:00.264758 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:00.264876 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:00.265409 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:10:00.265426 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:10:00.266800 (+ 1374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:10:00.268560 (+ 1760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:10:00.268566 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:10:00.268568 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:10:00.268796 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:10:00.268958 (+ 162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:10:00.268959 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:10:00.268959 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:10:00.269051 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:10:00.269455 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:10:00.269461 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:10:00.269596 (+ 135us) client_negotiation.cc:769] Sending connection context
0926 01:10:00.269697 (+ 101us) client_negotiation.cc:240] Negotiation successful
0926 01:10:00.269988 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":92,"threads_started":1}
I20260926 01:10:00.270677 1988 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42075
I20260926 01:10:00.270769 1988 heartbeater.cc:464] Registering TS with master...
I20260926 01:10:00.270951 1988 heartbeater.cc:511] Master 127.0.98.62:42075 requested a full tablet report, sending...
I20260926 01:10:00.271354 1503 ts_manager.cc:194] Registered new tserver with Master: caf1cf8c0a49436fb92743a2cddc285c (127.0.98.3:42925)
I20260926 01:10:00.271729 1503 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:57095
I20260926 01:10:00.277004 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 1856
I20260926 01:10:00.277073 1994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:00.271719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42075 (local address 127.0.98.3:40639)
0926 01:10:00.271844 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:00.271855 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:00.271901 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:00.272060 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:00.272061 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:00.272269 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:10:00.272337 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:00.272339 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:00.273040 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:00.273041 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:00.273569 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:00.273571 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:00.273978 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:00.273978 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:00.274027 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:00.274453 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:10:00.274457 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:10:00.274747 (+ 290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:10:00.276150 (+ 1403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:10:00.276151 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:10:00.276152 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:10:00.276309 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:10:00.276521 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:10:00.276522 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:10:00.276523 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:10:00.276592 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:10:00.276789 (+ 197us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:10:00.276790 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:10:00.276851 (+ 61us) client_negotiation.cc:769] Sending connection context
0926 01:10:00.276897 (+ 46us) client_negotiation.cc:240] Negotiation successful
0926 01:10:00.276980 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":60,"threads_started":1}
I20260926 01:10:00.277112 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384993574762-392-0/minicluster-data/ts-2/wal/instance
I20260926 01:10:00.277129 1880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:00.271809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40639 (local address 127.0.98.62:42075)
0926 01:10:00.271931 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:00.271933 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:00.271942 (+ 9us) server_negotiation.cc:407] Connection header received
0926 01:10:00.271961 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:00.271961 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:00.271982 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:00.272055 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:00.272412 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:00.272943 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:00.273667 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:00.273921 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:00.273978 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:00.274828 (+ 850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:10:00.274829 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:10:00.274829 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:10:00.274835 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:10:00.276048 (+ 1213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:00.276386 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:00.276387 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:00.276387 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:00.276439 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:00.276651 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:00.276651 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:00.276652 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:00.276750 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:10:00.276793 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:00.276934 (+ 141us) server_negotiation.cc:299] Negotiation successful
0926 01:10:00.277016 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260926 01:10:00.277290 392 external_mini_cluster.cc:1428] Setting key aa8d6e4b3a09963cb4319e4c06897c42
I20260926 01:10:00.278115 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 01:10:00 dist-test-slave-7b22 krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385000, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 01:10:00 dist-test-slave-7b22 krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385000, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:10:00.308457 1880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:00.301050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38136 (local address 127.0.98.62:42075)
0926 01:10:00.301097 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:00.301099 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:00.301110 (+ 11us) server_negotiation.cc:407] Connection header received
0926 01:10:00.301157 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:00.301158 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:00.301170 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:00.301251 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:00.302078 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:00.302839 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:00.303793 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:00.303983 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:00.304034 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:00.306199 (+ 2165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:10:00.306201 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:10:00.306201 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:10:00.306210 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:10:00.307472 (+ 1262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:00.307797 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:00.307798 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:00.307798 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:00.307848 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:00.307975 (+ 127us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:00.307976 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:00.307976 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:00.308074 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:10:00.308146 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:00.308326 (+ 180us) server_negotiation.cc:299] Negotiation successful
0926 01:10:00.308387 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 01:10:00.310021 1503 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38136:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:10:00.311517 1503 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:10:00.318120 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:00.314160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42925 (local address 127.0.0.1:56922)
0926 01:10:00.314315 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:00.314326 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:00.314384 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:00.314620 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:00.314621 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:00.314623 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:10:00.314658 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:00.314659 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:00.315922 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:00.315923 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:00.316918 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:00.316921 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:00.317760 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:00.317761 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:00.317860 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:00.317879 (+ 19us) client_negotiation.cc:769] Sending connection context
0926 01:10:00.317969 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 01:10:00.318031 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":76,"threads_started":1}
I20260926 01:10:00.318140 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:00.314268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56922 (local address 127.0.98.3:42925)
0926 01:10:00.314435 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:00.314439 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:00.314449 (+ 10us) server_negotiation.cc:407] Connection header received
0926 01:10:00.314483 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:00.314485 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:00.314498 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:00.314563 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:10:00.314742 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:00.315830 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:00.317008 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:00.317676 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:00.317750 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:00.317859 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:00.317987 (+ 128us) server_negotiation.cc:299] Negotiation successful
0926 01:10:00.318026 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":93,"threads_started":1}
I20260926 01:10:00.318490 2005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:00.313919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37623 (local address 127.0.0.1:53726)
0926 01:10:00.314426 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:00.314436 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:00.314516 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:00.314797 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:00.314798 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:00.314800 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:10:00.314843 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:00.314844 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:00.315770 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:00.315771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:00.317154 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:00.317157 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:00.318274 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:00.318275 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:00.318327 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:00.318331 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 01:10:00.318378 (+ 47us) client_negotiation.cc:240] Negotiation successful
0926 01:10:00.318420 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"mutex_wait_us":32,"thread_start_us":116,"threads_started":1}
I20260926 01:10:00.318516 2007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:00.313997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53726 (local address 127.0.98.2:37623)
0926 01:10:00.314170 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:00.314175 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:00.314490 (+ 315us) server_negotiation.cc:407] Connection header received
0926 01:10:00.314607 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:00.314610 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:00.314627 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:00.314700 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:10:00.314910 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:00.315680 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:00.317246 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:00.318114 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:00.318181 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:00.318240 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:00.318404 (+ 164us) server_negotiation.cc:299] Negotiation successful
0926 01:10:00.318446 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":47,"threads_started":1}
I20260926 01:10:00.319216 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:00.314003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59704 (local address 127.0.98.1:42393)
0926 01:10:00.314233 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:00.314236 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:00.314248 (+ 12us) server_negotiation.cc:407] Connection header received
0926 01:10:00.314339 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:00.314343 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:00.314358 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:00.314488 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:10:00.314726 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:00.315820 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:00.317923 (+ 2103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:00.318734 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:00.318804 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:00.318862 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:00.319085 (+ 223us) server_negotiation.cc:299] Negotiation successful
0926 01:10:00.319131 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":60,"threads_started":1}
I20260926 01:10:00.319259 1922 tablet_service.cc:1467] Processing CreateTablet for tablet c4ae903809a14658ae33331470017267 (DEFAULT_TABLE table=test-table [id=ae365c1ef3d849ab9afa3398a00c6bda]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:10:00.319624 1922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4ae903809a14658ae33331470017267. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:10:00.319139 2006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:00.313895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42393 (local address 127.0.0.1:59704)
0926 01:10:00.314145 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:00.314187 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:00.314247 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:00.314571 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:00.314574 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:00.314577 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:10:00.314645 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:00.314648 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:00.315904 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:00.315905 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:00.317155 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:00.317157 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:00.318895 (+ 1738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:00.318896 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:00.318969 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:00.318985 (+ 16us) client_negotiation.cc:769] Sending connection context
0926 01:10:00.319037 (+ 52us) client_negotiation.cc:240] Negotiation successful
0926 01:10:00.319081 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"mutex_wait_us":5,"thread_start_us":113,"threads_started":1}
I20260926 01:10:00.322623 2011 tablet_bootstrap.cc:492] T c4ae903809a14658ae33331470017267 P caf1cf8c0a49436fb92743a2cddc285c: Bootstrap starting.
I20260926 01:10:00.322582 1781 tablet_service.cc:1467] Processing CreateTablet for tablet c4ae903809a14658ae33331470017267 (DEFAULT_TABLE table=test-table [id=ae365c1ef3d849ab9afa3398a00c6bda]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:10:00.322736 1641 tablet_service.cc:1467] Processing CreateTablet for tablet c4ae903809a14658ae33331470017267 (DEFAULT_TABLE table=test-table [id=ae365c1ef3d849ab9afa3398a00c6bda]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:10:00.322937 1781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4ae903809a14658ae33331470017267. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:10:00.323089 1641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4ae903809a14658ae33331470017267. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:10:00.323352 2011 tablet_bootstrap.cc:654] T c4ae903809a14658ae33331470017267 P caf1cf8c0a49436fb92743a2cddc285c: Neither blocks nor log segments found. Creating new log.
I20260926 01:10:00.323688 2011 log.cc:824] T c4ae903809a14658ae33331470017267 P caf1cf8c0a49436fb92743a2cddc285c: Log is configured to *not* fsync() on all Append() calls
I20260926 01:10:00.324536 2011 tablet_bootstrap.cc:492] T c4ae903809a14658ae33331470017267 P caf1cf8c0a49436fb92743a2cddc285c: No bootstrap required, opened a new log
I20260926 01:10:00.324600 2011 ts_tablet_manager.cc:1397] T c4ae903809a14658ae33331470017267 P caf1cf8c0a49436fb92743a2cddc285c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 01:10:00.325942 2013 tablet_bootstrap.cc:492] T c4ae903809a14658ae33331470017267 P f873f7c4a5944baab02c621fee691375: Bootstrap starting.
I20260926 01:10:00.326100 2014 tablet_bootstrap.cc:492] T c4ae903809a14658ae33331470017267 P 04f5f856ae004e06bfe8d74881ec7371: Bootstrap starting.
I20260926 01:10:00.326696 2013 tablet_bootstrap.cc:654] T c4ae903809a14658ae33331470017267 P f873f7c4a5944baab02c621fee691375: Neither blocks nor log segments found. Creating new log.
I20260926 01:10:00.326696 2014 tablet_bootstrap.cc:654] T c4ae903809a14658ae33331470017267 P 04f5f856ae004e06bfe8d74881ec7371: Neither blocks nor log segments found. Creating new log.
I20260926 01:10:00.326656 2011 raft_consensus.cc:348] T c4ae903809a14658ae33331470017267 P caf1cf8c0a49436fb92743a2cddc285c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04f5f856ae004e06bfe8d74881ec7371" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42393 } } peers { permanent_uuid: "caf1cf8c0a49436fb92743a2cddc285c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42925 } } peers { permanent_uuid: "f873f7c4a5944baab02c621fee691375" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37623 } }
I20260926 01:10:00.326789 2011 raft_consensus.cc:374] T c4ae903809a14658ae33331470017267 P caf1cf8c0a49436fb92743a2cddc285c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:10:00.326843 2011 raft_consensus.cc:729] T c4ae903809a14658ae33331470017267 P caf1cf8c0a49436fb92743a2cddc285c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: caf1cf8c0a49436fb92743a2cddc285c, State: Initialized, Role: FOLLOWER
I20260926 01:10:00.326970 2011 consensus_queue.cc:260] T c4ae903809a14658ae33331470017267 P caf1cf8c0a49436fb92743a2cddc285c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04f5f856ae004e06bfe8d74881ec7371" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42393 } } peers { permanent_uuid: "caf1cf8c0a49436fb92743a2cddc285c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42925 } } peers { permanent_uuid: "f873f7c4a5944baab02c621fee691375" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37623 } }
I20260926 01:10:00.327008 2013 log.cc:824] T c4ae903809a14658ae33331470017267 P f873f7c4a5944baab02c621fee691375: Log is configured to *not* fsync() on all Append() calls
I20260926 01:10:00.327008 2014 log.cc:824] T c4ae903809a14658ae33331470017267 P 04f5f856ae004e06bfe8d74881ec7371: Log is configured to *not* fsync() on all Append() calls
I20260926 01:10:00.327241 2011 ts_tablet_manager.cc:1428] T c4ae903809a14658ae33331470017267 P caf1cf8c0a49436fb92743a2cddc285c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 01:10:00.327389 1988 heartbeater.cc:503] Master 127.0.98.62:42075 was elected leader, sending a full tablet report...
I20260926 01:10:00.327682 2013 tablet_bootstrap.cc:492] T c4ae903809a14658ae33331470017267 P f873f7c4a5944baab02c621fee691375: No bootstrap required, opened a new log
I20260926 01:10:00.327859 2014 tablet_bootstrap.cc:492] T c4ae903809a14658ae33331470017267 P 04f5f856ae004e06bfe8d74881ec7371: No bootstrap required, opened a new log
I20260926 01:10:00.327916 2014 ts_tablet_manager.cc:1397] T c4ae903809a14658ae33331470017267 P 04f5f856ae004e06bfe8d74881ec7371: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 01:10:00.328053 2013 ts_tablet_manager.cc:1397] T c4ae903809a14658ae33331470017267 P f873f7c4a5944baab02c621fee691375: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 01:10:00.329985 2014 raft_consensus.cc:348] T c4ae903809a14658ae33331470017267 P 04f5f856ae004e06bfe8d74881ec7371 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04f5f856ae004e06bfe8d74881ec7371" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42393 } } peers { permanent_uuid: "caf1cf8c0a49436fb92743a2cddc285c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42925 } } peers { permanent_uuid: "f873f7c4a5944baab02c621fee691375" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37623 } }
I20260926 01:10:00.330118 2014 raft_consensus.cc:374] T c4ae903809a14658ae33331470017267 P 04f5f856ae004e06bfe8d74881ec7371 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:10:00.330138 2014 raft_consensus.cc:729] T c4ae903809a14658ae33331470017267 P 04f5f856ae004e06bfe8d74881ec7371 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04f5f856ae004e06bfe8d74881ec7371, State: Initialized, Role: FOLLOWER
I20260926 01:10:00.330240 2014 consensus_queue.cc:260] T c4ae903809a14658ae33331470017267 P 04f5f856ae004e06bfe8d74881ec7371 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04f5f856ae004e06bfe8d74881ec7371" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42393 } } peers { permanent_uuid: "caf1cf8c0a49436fb92743a2cddc285c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42925 } } peers { permanent_uuid: "f873f7c4a5944baab02c621fee691375" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37623 } }
I20260926 01:10:00.330461 2014 ts_tablet_manager.cc:1428] T c4ae903809a14658ae33331470017267 P 04f5f856ae004e06bfe8d74881ec7371: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 01:10:00.330472 1707 heartbeater.cc:503] Master 127.0.98.62:42075 was elected leader, sending a full tablet report...
I20260926 01:10:00.330430 2013 raft_consensus.cc:348] T c4ae903809a14658ae33331470017267 P f873f7c4a5944baab02c621fee691375 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04f5f856ae004e06bfe8d74881ec7371" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42393 } } peers { permanent_uuid: "caf1cf8c0a49436fb92743a2cddc285c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42925 } } peers { permanent_uuid: "f873f7c4a5944baab02c621fee691375" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37623 } }
I20260926 01:10:00.330557 2013 raft_consensus.cc:374] T c4ae903809a14658ae33331470017267 P f873f7c4a5944baab02c621fee691375 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:10:00.330588 2013 raft_consensus.cc:729] T c4ae903809a14658ae33331470017267 P f873f7c4a5944baab02c621fee691375 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f873f7c4a5944baab02c621fee691375, State: Initialized, Role: FOLLOWER
I20260926 01:10:00.330693 2013 consensus_queue.cc:260] T c4ae903809a14658ae33331470017267 P f873f7c4a5944baab02c621fee691375 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04f5f856ae004e06bfe8d74881ec7371" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42393 } } peers { permanent_uuid: "caf1cf8c0a49436fb92743a2cddc285c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42925 } } peers { permanent_uuid: "f873f7c4a5944baab02c621fee691375" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37623 } }
I20260926 01:10:00.330940 2013 ts_tablet_manager.cc:1428] T c4ae903809a14658ae33331470017267 P f873f7c4a5944baab02c621fee691375: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260926 01:10:00.330967 1847 heartbeater.cc:503] Master 127.0.98.62:42075 was elected leader, sending a full tablet report...
W20260926 01:10:00.332635 1708 tablet.cc:2369] T c4ae903809a14658ae33331470017267 P 04f5f856ae004e06bfe8d74881ec7371: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 01:10:00.473441 2018 raft_consensus.cc:482] T c4ae903809a14658ae33331470017267 P 04f5f856ae004e06bfe8d74881ec7371 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:10:00.473578 2018 raft_consensus.cc:504] T c4ae903809a14658ae33331470017267 P 04f5f856ae004e06bfe8d74881ec7371 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04f5f856ae004e06bfe8d74881ec7371" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42393 } } peers { permanent_uuid: "caf1cf8c0a49436fb92743a2cddc285c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42925 } } peers { permanent_uuid: "f873f7c4a5944baab02c621fee691375" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37623 } }
I20260926 01:10:00.473946 2018 leader_election.cc:288] T c4ae903809a14658ae33331470017267 P 04f5f856ae004e06bfe8d74881ec7371 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers caf1cf8c0a49436fb92743a2cddc285c (127.0.98.3:42925), f873f7c4a5944baab02c621fee691375 (127.0.98.2:37623)
I20260926 01:10:00.478137 2007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:00.474232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58761 (local address 127.0.98.2:37623)
0926 01:10:00.474278 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:00.474281 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:00.474511 (+ 230us) server_negotiation.cc:407] Connection header received
0926 01:10:00.474547 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:00.474549 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:00.474564 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:00.474632 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:10:00.474851 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:00.475610 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:00.477141 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:00.477712 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:00.477804 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:00.477840 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:00.478017 (+ 177us) server_negotiation.cc:299] Negotiation successful
0926 01:10:00.478058 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 01:10:00.478135 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:00.474268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55291 (local address 127.0.98.3:42925)
0926 01:10:00.474318 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:00.474320 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:00.474429 (+ 109us) server_negotiation.cc:407] Connection header received
0926 01:10:00.474567 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:00.474567 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:00.474580 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:00.474632 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:10:00.474852 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:00.475590 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:00.477140 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:00.477749 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:00.477808 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:00.477841 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:00.478017 (+ 176us) server_negotiation.cc:299] Negotiation successful
0926 01:10:00.478054 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 01:10:00.478202 2020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:00.474205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37623 (local address 127.0.98.1:58761)
0926 01:10:00.474437 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:00.474445 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:00.474492 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:00.474653 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:00.474654 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:00.474655 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:10:00.474756 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:00.474759 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:00.475680 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:00.475681 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:00.477017 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:00.477020 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:00.477851 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:00.477852 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:00.477939 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:00.477946 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 01:10:00.478007 (+ 61us) client_negotiation.cc:240] Negotiation successful
0926 01:10:00.478062 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"mutex_wait_us":102,"thread_start_us":97,"threads_started":1}
I20260926 01:10:00.478202 2021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:00.474166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42925 (local address 127.0.98.1:55291)
0926 01:10:00.474353 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:00.474365 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:00.474434 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:00.474651 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:00.474652 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:00.474654 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:10:00.474756 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:00.474758 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:00.475681 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:00.475682 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:00.477017 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:00.477020 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:00.477852 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:00.477853 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:00.477939 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:00.477947 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 01:10:00.478007 (+ 60us) client_negotiation.cc:240] Negotiation successful
0926 01:10:00.478059 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"mutex_wait_us":36,"thread_start_us":114,"threads_started":1}
I20260926 01:10:00.478567 1801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4ae903809a14658ae33331470017267" candidate_uuid: "04f5f856ae004e06bfe8d74881ec7371" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f873f7c4a5944baab02c621fee691375" is_pre_election: true
I20260926 01:10:00.478734 1801 raft_consensus.cc:2454] T c4ae903809a14658ae33331470017267 P f873f7c4a5944baab02c621fee691375 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 04f5f856ae004e06bfe8d74881ec7371 in term 0.
I20260926 01:10:00.478691 1942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4ae903809a14658ae33331470017267" candidate_uuid: "04f5f856ae004e06bfe8d74881ec7371" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "caf1cf8c0a49436fb92743a2cddc285c" is_pre_election: true
I20260926 01:10:00.478849 1942 raft_consensus.cc:2454] T c4ae903809a14658ae33331470017267 P caf1cf8c0a49436fb92743a2cddc285c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 04f5f856ae004e06bfe8d74881ec7371 in term 0.
I20260926 01:10:00.478951 1593 leader_election.cc:302] T c4ae903809a14658ae33331470017267 P 04f5f856ae004e06bfe8d74881ec7371 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04f5f856ae004e06bfe8d74881ec7371, f873f7c4a5944baab02c621fee691375; no voters:
I20260926 01:10:00.479076 2018 raft_consensus.cc:2781] T c4ae903809a14658ae33331470017267 P 04f5f856ae004e06bfe8d74881ec7371 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:10:00.479151 2018 raft_consensus.cc:482] T c4ae903809a14658ae33331470017267 P 04f5f856ae004e06bfe8d74881ec7371 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:10:00.479172 2018 raft_consensus.cc:3037] T c4ae903809a14658ae33331470017267 P 04f5f856ae004e06bfe8d74881ec7371 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:10:00.479904 2018 raft_consensus.cc:504] T c4ae903809a14658ae33331470017267 P 04f5f856ae004e06bfe8d74881ec7371 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04f5f856ae004e06bfe8d74881ec7371" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42393 } } peers { permanent_uuid: "caf1cf8c0a49436fb92743a2cddc285c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42925 } } peers { permanent_uuid: "f873f7c4a5944baab02c621fee691375" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37623 } }
I20260926 01:10:00.480053 2018 leader_election.cc:288] T c4ae903809a14658ae33331470017267 P 04f5f856ae004e06bfe8d74881ec7371 [CANDIDATE]: Term 1 election: Requested vote from peers caf1cf8c0a49436fb92743a2cddc285c (127.0.98.3:42925), f873f7c4a5944baab02c621fee691375 (127.0.98.2:37623)
I20260926 01:10:00.480211 1942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4ae903809a14658ae33331470017267" candidate_uuid: "04f5f856ae004e06bfe8d74881ec7371" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "caf1cf8c0a49436fb92743a2cddc285c"
I20260926 01:10:00.480284 1942 raft_consensus.cc:3037] T c4ae903809a14658ae33331470017267 P caf1cf8c0a49436fb92743a2cddc285c [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:10:00.480288 1801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4ae903809a14658ae33331470017267" candidate_uuid: "04f5f856ae004e06bfe8d74881ec7371" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f873f7c4a5944baab02c621fee691375"
I20260926 01:10:00.480393 1801 raft_consensus.cc:3037] T c4ae903809a14658ae33331470017267 P f873f7c4a5944baab02c621fee691375 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:10:00.480989 1942 raft_consensus.cc:2454] T c4ae903809a14658ae33331470017267 P caf1cf8c0a49436fb92743a2cddc285c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 04f5f856ae004e06bfe8d74881ec7371 in term 1.
I20260926 01:10:00.481125 1801 raft_consensus.cc:2454] T c4ae903809a14658ae33331470017267 P f873f7c4a5944baab02c621fee691375 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 04f5f856ae004e06bfe8d74881ec7371 in term 1.
I20260926 01:10:00.481160 1592 leader_election.cc:302] T c4ae903809a14658ae33331470017267 P 04f5f856ae004e06bfe8d74881ec7371 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04f5f856ae004e06bfe8d74881ec7371, caf1cf8c0a49436fb92743a2cddc285c; no voters:
I20260926 01:10:00.481299 2018 raft_consensus.cc:2781] T c4ae903809a14658ae33331470017267 P 04f5f856ae004e06bfe8d74881ec7371 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:10:00.481459 2018 raft_consensus.cc:686] T c4ae903809a14658ae33331470017267 P 04f5f856ae004e06bfe8d74881ec7371 [term 1 LEADER]: Becoming Leader. State: Replica: 04f5f856ae004e06bfe8d74881ec7371, State: Running, Role: LEADER
I20260926 01:10:00.481550 2018 consensus_queue.cc:237] T c4ae903809a14658ae33331470017267 P 04f5f856ae004e06bfe8d74881ec7371 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04f5f856ae004e06bfe8d74881ec7371" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42393 } } peers { permanent_uuid: "caf1cf8c0a49436fb92743a2cddc285c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42925 } } peers { permanent_uuid: "f873f7c4a5944baab02c621fee691375" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37623 } }
I20260926 01:10:00.482396 1503 catalog_manager.cc:5495] T c4ae903809a14658ae33331470017267 P 04f5f856ae004e06bfe8d74881ec7371 reported cstate change: term changed from 0 to 1, leader changed from <none> to 04f5f856ae004e06bfe8d74881ec7371 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "04f5f856ae004e06bfe8d74881ec7371" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04f5f856ae004e06bfe8d74881ec7371" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42393 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "caf1cf8c0a49436fb92743a2cddc285c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42925 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f873f7c4a5944baab02c621fee691375" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37623 } health_report { overall_health: UNKNOWN } } }
W20260926 01:10:00.485267 1848 tablet.cc:2369] T c4ae903809a14658ae33331470017267 P f873f7c4a5944baab02c621fee691375: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 01:10:00.500674 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:00.497301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59706 (local address 127.0.98.1:42393)
0926 01:10:00.497352 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:00.497355 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:00.497384 (+ 29us) server_negotiation.cc:407] Connection header received
0926 01:10:00.497426 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:00.497427 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:00.497453 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:00.497523 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:10:00.497700 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:00.498525 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:00.499684 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:00.499988 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:00.500057 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:00.500309 (+ 252us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:10:00.500431 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:10:00.500503 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:00.500570 (+ 67us) server_negotiation.cc:299] Negotiation successful
0926 01:10:00.500594 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 01:10:00.504244 1942 raft_consensus.cc:1261] T c4ae903809a14658ae33331470017267 P caf1cf8c0a49436fb92743a2cddc285c [term 1 FOLLOWER]: Refusing update from remote peer 04f5f856ae004e06bfe8d74881ec7371: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:10:00.504583 2022 consensus_queue.cc:1035] T c4ae903809a14658ae33331470017267 P 04f5f856ae004e06bfe8d74881ec7371 [LEADER]: Connected to new peer: Peer: permanent_uuid: "caf1cf8c0a49436fb92743a2cddc285c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:10:00.505335 1801 raft_consensus.cc:1261] T c4ae903809a14658ae33331470017267 P f873f7c4a5944baab02c621fee691375 [term 1 FOLLOWER]: Refusing update from remote peer 04f5f856ae004e06bfe8d74881ec7371: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:10:00.507035 2025 mvcc.cc:205] Tried to move back new op lower bound from 7333416962062086144 to 7333416961972957184. Current Snapshot: MvccSnapshot[applied={T|T < 7333416962062086144}]
I20260926 01:10:00.507408 2022 consensus_queue.cc:1035] T c4ae903809a14658ae33331470017267 P 04f5f856ae004e06bfe8d74881ec7371 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f873f7c4a5944baab02c621fee691375" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:10:00.510978 1503 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38136:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:10:00.511073 1503 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38136:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:10:00.511739 1503 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a3fc3ff0f3ad4ec19518afe126ee308f: Sending DeleteTablet for 3 replicas of tablet c4ae903809a14658ae33331470017267
I20260926 01:10:00.512037 1641 tablet_service.cc:1514] Processing DeleteTablet for tablet c4ae903809a14658ae33331470017267 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:10:00 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59704
I20260926 01:10:00.512091 1922 tablet_service.cc:1514] Processing DeleteTablet for tablet c4ae903809a14658ae33331470017267 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:10:00 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56922
I20260926 01:10:00.512301 2034 tablet_replica.cc:330] T c4ae903809a14658ae33331470017267 P 04f5f856ae004e06bfe8d74881ec7371: stopping tablet replica
I20260926 01:10:00.512295 1781 tablet_service.cc:1514] Processing DeleteTablet for tablet c4ae903809a14658ae33331470017267 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:10:00 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53726
I20260926 01:10:00.512387 1503 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38136:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 01:10:00.512477 2034 raft_consensus.cc:2229] T c4ae903809a14658ae33331470017267 P 04f5f856ae004e06bfe8d74881ec7371 [term 1 LEADER]: Raft consensus shutting down.
I20260926 01:10:00.512516 2035 tablet_replica.cc:330] T c4ae903809a14658ae33331470017267 P caf1cf8c0a49436fb92743a2cddc285c: stopping tablet replica
W20260926 01:10:00.512530 1503 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:10:00.512624 2034 raft_consensus.cc:2258] T c4ae903809a14658ae33331470017267 P 04f5f856ae004e06bfe8d74881ec7371 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:10:00.512655 2035 raft_consensus.cc:2229] T c4ae903809a14658ae33331470017267 P caf1cf8c0a49436fb92743a2cddc285c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:10:00.512765 2035 raft_consensus.cc:2258] T c4ae903809a14658ae33331470017267 P caf1cf8c0a49436fb92743a2cddc285c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:10:00.513064 2034 ts_tablet_manager.cc:1905] T c4ae903809a14658ae33331470017267 P 04f5f856ae004e06bfe8d74881ec7371: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:10:00.513162 2035 ts_tablet_manager.cc:1905] T c4ae903809a14658ae33331470017267 P caf1cf8c0a49436fb92743a2cddc285c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:10:00.514110 1781 tablet_service.cc:1467] Processing CreateTablet for tablet 801368c3811a450a9f0aefacd9cd2f79 (DEFAULT_TABLE table=test-table [id=3f17b7aee917453387e105b9a01e1a79]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:10:00.514225 1781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 801368c3811a450a9f0aefacd9cd2f79. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:10:00.514915 2034 ts_tablet_manager.cc:1918] T c4ae903809a14658ae33331470017267 P 04f5f856ae004e06bfe8d74881ec7371: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:10:00.514974 2034 log.cc:1196] T c4ae903809a14658ae33331470017267 P 04f5f856ae004e06bfe8d74881ec7371: Deleting WAL directory at /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384993574762-392-0/minicluster-data/ts-0/wal/wals/c4ae903809a14658ae33331470017267
I20260926 01:10:00.515102 2035 ts_tablet_manager.cc:1918] T c4ae903809a14658ae33331470017267 P caf1cf8c0a49436fb92743a2cddc285c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:10:00.515164 2035 log.cc:1196] T c4ae903809a14658ae33331470017267 P caf1cf8c0a49436fb92743a2cddc285c: Deleting WAL directory at /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384993574762-392-0/minicluster-data/ts-2/wal/wals/c4ae903809a14658ae33331470017267
I20260926 01:10:00.515192 2034 ts_tablet_manager.cc:1939] T c4ae903809a14658ae33331470017267 P 04f5f856ae004e06bfe8d74881ec7371: Deleting consensus metadata
I20260926 01:10:00.515394 2035 ts_tablet_manager.cc:1939] T c4ae903809a14658ae33331470017267 P caf1cf8c0a49436fb92743a2cddc285c: Deleting consensus metadata
I20260926 01:10:00.515522 2013 tablet_bootstrap.cc:492] T 801368c3811a450a9f0aefacd9cd2f79 P f873f7c4a5944baab02c621fee691375: Bootstrap starting.
I20260926 01:10:00.515848 2013 tablet_bootstrap.cc:654] T 801368c3811a450a9f0aefacd9cd2f79 P f873f7c4a5944baab02c621fee691375: Neither blocks nor log segments found. Creating new log.
I20260926 01:10:00.516426 1488 catalog_manager.cc:4841] TS caf1cf8c0a49436fb92743a2cddc285c (127.0.98.3:42925): tablet c4ae903809a14658ae33331470017267 (table test-table [id=ae365c1ef3d849ab9afa3398a00c6bda]) successfully deleted
I20260926 01:10:00.516515 1488 catalog_manager.cc:4841] TS 04f5f856ae004e06bfe8d74881ec7371 (127.0.98.1:42393): tablet c4ae903809a14658ae33331470017267 (table test-table [id=ae365c1ef3d849ab9afa3398a00c6bda]) successfully deleted
I20260926 01:10:00.516513 1641 tablet_service.cc:1467] Processing CreateTablet for tablet 801368c3811a450a9f0aefacd9cd2f79 (DEFAULT_TABLE table=test-table [id=3f17b7aee917453387e105b9a01e1a79]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:10:00.516598 1641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 801368c3811a450a9f0aefacd9cd2f79. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:10:00.516574 1922 tablet_service.cc:1467] Processing CreateTablet for tablet 801368c3811a450a9f0aefacd9cd2f79 (DEFAULT_TABLE table=test-table [id=3f17b7aee917453387e105b9a01e1a79]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:10:00.516695 1922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 801368c3811a450a9f0aefacd9cd2f79. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:10:00.516881 2013 tablet_bootstrap.cc:492] T 801368c3811a450a9f0aefacd9cd2f79 P f873f7c4a5944baab02c621fee691375: No bootstrap required, opened a new log
I20260926 01:10:00.516927 2013 ts_tablet_manager.cc:1397] T 801368c3811a450a9f0aefacd9cd2f79 P f873f7c4a5944baab02c621fee691375: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260926 01:10:00.517091 2013 raft_consensus.cc:348] T 801368c3811a450a9f0aefacd9cd2f79 P f873f7c4a5944baab02c621fee691375 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04f5f856ae004e06bfe8d74881ec7371" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42393 } } peers { permanent_uuid: "f873f7c4a5944baab02c621fee691375" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37623 } } peers { permanent_uuid: "caf1cf8c0a49436fb92743a2cddc285c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42925 } }
I20260926 01:10:00.517153 2013 raft_consensus.cc:374] T 801368c3811a450a9f0aefacd9cd2f79 P f873f7c4a5944baab02c621fee691375 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:10:00.517172 2013 raft_consensus.cc:729] T 801368c3811a450a9f0aefacd9cd2f79 P f873f7c4a5944baab02c621fee691375 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f873f7c4a5944baab02c621fee691375, State: Initialized, Role: FOLLOWER
I20260926 01:10:00.517211 2013 consensus_queue.cc:260] T 801368c3811a450a9f0aefacd9cd2f79 P f873f7c4a5944baab02c621fee691375 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04f5f856ae004e06bfe8d74881ec7371" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42393 } } peers { permanent_uuid: "f873f7c4a5944baab02c621fee691375" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37623 } } peers { permanent_uuid: "caf1cf8c0a49436fb92743a2cddc285c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42925 } }
I20260926 01:10:00.517306 2013 ts_tablet_manager.cc:1428] T 801368c3811a450a9f0aefacd9cd2f79 P f873f7c4a5944baab02c621fee691375: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:00.517767 2014 tablet_bootstrap.cc:492] T 801368c3811a450a9f0aefacd9cd2f79 P 04f5f856ae004e06bfe8d74881ec7371: Bootstrap starting.
I20260926 01:10:00.518189 2014 tablet_bootstrap.cc:654] T 801368c3811a450a9f0aefacd9cd2f79 P 04f5f856ae004e06bfe8d74881ec7371: Neither blocks nor log segments found. Creating new log.
I20260926 01:10:00.518230 2036 tablet_replica.cc:330] T c4ae903809a14658ae33331470017267 P f873f7c4a5944baab02c621fee691375: stopping tablet replica
I20260926 01:10:00.518342 2036 raft_consensus.cc:2229] T c4ae903809a14658ae33331470017267 P f873f7c4a5944baab02c621fee691375 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:10:00.518437 2036 raft_consensus.cc:2258] T c4ae903809a14658ae33331470017267 P f873f7c4a5944baab02c621fee691375 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:10:00.518630 2011 tablet_bootstrap.cc:492] T 801368c3811a450a9f0aefacd9cd2f79 P caf1cf8c0a49436fb92743a2cddc285c: Bootstrap starting.
I20260926 01:10:00.518903 2014 tablet_bootstrap.cc:492] T 801368c3811a450a9f0aefacd9cd2f79 P 04f5f856ae004e06bfe8d74881ec7371: No bootstrap required, opened a new log
I20260926 01:10:00.518944 2014 ts_tablet_manager.cc:1397] T 801368c3811a450a9f0aefacd9cd2f79 P 04f5f856ae004e06bfe8d74881ec7371: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 01:10:00.519069 2014 raft_consensus.cc:348] T 801368c3811a450a9f0aefacd9cd2f79 P 04f5f856ae004e06bfe8d74881ec7371 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04f5f856ae004e06bfe8d74881ec7371" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42393 } } peers { permanent_uuid: "f873f7c4a5944baab02c621fee691375" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37623 } } peers { permanent_uuid: "caf1cf8c0a49436fb92743a2cddc285c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42925 } }
I20260926 01:10:00.519125 2014 raft_consensus.cc:374] T 801368c3811a450a9f0aefacd9cd2f79 P 04f5f856ae004e06bfe8d74881ec7371 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:10:00.519142 2014 raft_consensus.cc:729] T 801368c3811a450a9f0aefacd9cd2f79 P 04f5f856ae004e06bfe8d74881ec7371 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04f5f856ae004e06bfe8d74881ec7371, State: Initialized, Role: FOLLOWER
I20260926 01:10:00.519187 2014 consensus_queue.cc:260] T 801368c3811a450a9f0aefacd9cd2f79 P 04f5f856ae004e06bfe8d74881ec7371 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04f5f856ae004e06bfe8d74881ec7371" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42393 } } peers { permanent_uuid: "f873f7c4a5944baab02c621fee691375" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37623 } } peers { permanent_uuid: "caf1cf8c0a49436fb92743a2cddc285c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42925 } }
I20260926 01:10:00.519289 2014 ts_tablet_manager.cc:1428] T 801368c3811a450a9f0aefacd9cd2f79 P 04f5f856ae004e06bfe8d74881ec7371: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:00.519538 2011 tablet_bootstrap.cc:654] T 801368c3811a450a9f0aefacd9cd2f79 P caf1cf8c0a49436fb92743a2cddc285c: Neither blocks nor log segments found. Creating new log.
I20260926 01:10:00.520193 2011 tablet_bootstrap.cc:492] T 801368c3811a450a9f0aefacd9cd2f79 P caf1cf8c0a49436fb92743a2cddc285c: No bootstrap required, opened a new log
I20260926 01:10:00.520246 2011 ts_tablet_manager.cc:1397] T 801368c3811a450a9f0aefacd9cd2f79 P caf1cf8c0a49436fb92743a2cddc285c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 01:10:00.520387 2011 raft_consensus.cc:348] T 801368c3811a450a9f0aefacd9cd2f79 P caf1cf8c0a49436fb92743a2cddc285c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04f5f856ae004e06bfe8d74881ec7371" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42393 } } peers { permanent_uuid: "f873f7c4a5944baab02c621fee691375" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37623 } } peers { permanent_uuid: "caf1cf8c0a49436fb92743a2cddc285c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42925 } }
I20260926 01:10:00.520453 2011 raft_consensus.cc:374] T 801368c3811a450a9f0aefacd9cd2f79 P caf1cf8c0a49436fb92743a2cddc285c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:10:00.520471 2011 raft_consensus.cc:729] T 801368c3811a450a9f0aefacd9cd2f79 P caf1cf8c0a49436fb92743a2cddc285c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: caf1cf8c0a49436fb92743a2cddc285c, State: Initialized, Role: FOLLOWER
I20260926 01:10:00.520507 2011 consensus_queue.cc:260] T 801368c3811a450a9f0aefacd9cd2f79 P caf1cf8c0a49436fb92743a2cddc285c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04f5f856ae004e06bfe8d74881ec7371" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42393 } } peers { permanent_uuid: "f873f7c4a5944baab02c621fee691375" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37623 } } peers { permanent_uuid: "caf1cf8c0a49436fb92743a2cddc285c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42925 } }
I20260926 01:10:00.520594 2011 ts_tablet_manager.cc:1428] T 801368c3811a450a9f0aefacd9cd2f79 P caf1cf8c0a49436fb92743a2cddc285c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:00.521060 2036 ts_tablet_manager.cc:1905] T c4ae903809a14658ae33331470017267 P f873f7c4a5944baab02c621fee691375: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:10:00.522632 2036 ts_tablet_manager.cc:1918] T c4ae903809a14658ae33331470017267 P f873f7c4a5944baab02c621fee691375: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:10:00.522708 2036 log.cc:1196] T c4ae903809a14658ae33331470017267 P f873f7c4a5944baab02c621fee691375: Deleting WAL directory at /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384993574762-392-0/minicluster-data/ts-1/wal/wals/c4ae903809a14658ae33331470017267
I20260926 01:10:00.522903 2036 ts_tablet_manager.cc:1939] T c4ae903809a14658ae33331470017267 P f873f7c4a5944baab02c621fee691375: Deleting consensus metadata
I20260926 01:10:00.523274 1489 catalog_manager.cc:4841] TS f873f7c4a5944baab02c621fee691375 (127.0.98.2:37623): tablet c4ae903809a14658ae33331470017267 (table test-table [id=ae365c1ef3d849ab9afa3398a00c6bda]) successfully deleted
I20260926 01:10:00.715108 2019 raft_consensus.cc:482] T 801368c3811a450a9f0aefacd9cd2f79 P f873f7c4a5944baab02c621fee691375 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:10:00.715204 2019 raft_consensus.cc:504] T 801368c3811a450a9f0aefacd9cd2f79 P f873f7c4a5944baab02c621fee691375 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04f5f856ae004e06bfe8d74881ec7371" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42393 } } peers { permanent_uuid: "f873f7c4a5944baab02c621fee691375" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37623 } } peers { permanent_uuid: "caf1cf8c0a49436fb92743a2cddc285c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42925 } }
I20260926 01:10:00.715490 2019 leader_election.cc:288] T 801368c3811a450a9f0aefacd9cd2f79 P f873f7c4a5944baab02c621fee691375 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 04f5f856ae004e06bfe8d74881ec7371 (127.0.98.1:42393), caf1cf8c0a49436fb92743a2cddc285c (127.0.98.3:42925)
I20260926 01:10:00.718536 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:00.715818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53869 (local address 127.0.98.1:42393)
0926 01:10:00.715888 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:00.715890 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:00.715942 (+ 52us) server_negotiation.cc:407] Connection header received
0926 01:10:00.716014 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:00.716016 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:00.716025 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:00.716075 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:10:00.716236 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:00.716719 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:00.717801 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:00.718215 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:00.718278 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:00.718289 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:00.718432 (+ 143us) server_negotiation.cc:299] Negotiation successful
0926 01:10:00.718471 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 01:10:00.718528 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:00.715976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53971 (local address 127.0.98.3:42925)
0926 01:10:00.716018 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:00.716020 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:00.716037 (+ 17us) server_negotiation.cc:407] Connection header received
0926 01:10:00.716063 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:00.716063 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:00.716072 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:00.716125 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:10:00.716264 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:00.716738 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:00.717841 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:00.718204 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:00.718263 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:00.718275 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:00.718414 (+ 139us) server_negotiation.cc:299] Negotiation successful
0926 01:10:00.718453 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 01:10:00.718587 2040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:00.715683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42393 (local address 127.0.98.2:53869)
0926 01:10:00.715845 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:00.715858 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:00.715952 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:00.716093 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:00.716094 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:00.716096 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:10:00.716175 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:00.716177 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:00.716808 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:00.716808 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:00.717705 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:00.717708 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:00.718312 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:00.718313 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:00.718362 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:00.718366 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 01:10:00.718414 (+ 48us) client_negotiation.cc:240] Negotiation successful
0926 01:10:00.718455 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"mutex_wait_us":41,"thread_start_us":79,"threads_started":1}
I20260926 01:10:00.718590 2041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:00.715857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42925 (local address 127.0.98.2:53971)
0926 01:10:00.715933 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:00.715942 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:00.715995 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:00.716136 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:00.716136 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:00.716138 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:10:00.716196 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:00.716197 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:00.716806 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:00.716808 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:00.717727 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:00.717730 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:00.718283 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:00.718284 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:00.718352 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:00.718355 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 01:10:00.718400 (+ 45us) client_negotiation.cc:240] Negotiation successful
0926 01:10:00.718447 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64,"mutex_wait_us":33,"thread_start_us":39,"threads_started":1}
I20260926 01:10:00.718932 1942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "801368c3811a450a9f0aefacd9cd2f79" candidate_uuid: "f873f7c4a5944baab02c621fee691375" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "caf1cf8c0a49436fb92743a2cddc285c" is_pre_election: true
I20260926 01:10:00.719019 1942 raft_consensus.cc:2454] T 801368c3811a450a9f0aefacd9cd2f79 P caf1cf8c0a49436fb92743a2cddc285c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f873f7c4a5944baab02c621fee691375 in term 0.
I20260926 01:10:00.719028 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "801368c3811a450a9f0aefacd9cd2f79" candidate_uuid: "f873f7c4a5944baab02c621fee691375" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04f5f856ae004e06bfe8d74881ec7371" is_pre_election: true
I20260926 01:10:00.719174 1661 raft_consensus.cc:2454] T 801368c3811a450a9f0aefacd9cd2f79 P 04f5f856ae004e06bfe8d74881ec7371 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f873f7c4a5944baab02c621fee691375 in term 0.
I20260926 01:10:00.719182 1732 leader_election.cc:302] T 801368c3811a450a9f0aefacd9cd2f79 P f873f7c4a5944baab02c621fee691375 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: caf1cf8c0a49436fb92743a2cddc285c, f873f7c4a5944baab02c621fee691375; no voters:
I20260926 01:10:00.719334 2019 raft_consensus.cc:2781] T 801368c3811a450a9f0aefacd9cd2f79 P f873f7c4a5944baab02c621fee691375 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:10:00.719379 2019 raft_consensus.cc:482] T 801368c3811a450a9f0aefacd9cd2f79 P f873f7c4a5944baab02c621fee691375 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:10:00.719393 2019 raft_consensus.cc:3037] T 801368c3811a450a9f0aefacd9cd2f79 P f873f7c4a5944baab02c621fee691375 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:10:00.720041 2019 raft_consensus.cc:504] T 801368c3811a450a9f0aefacd9cd2f79 P f873f7c4a5944baab02c621fee691375 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04f5f856ae004e06bfe8d74881ec7371" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42393 } } peers { permanent_uuid: "f873f7c4a5944baab02c621fee691375" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37623 } } peers { permanent_uuid: "caf1cf8c0a49436fb92743a2cddc285c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42925 } }
I20260926 01:10:00.720153 2019 leader_election.cc:288] T 801368c3811a450a9f0aefacd9cd2f79 P f873f7c4a5944baab02c621fee691375 [CANDIDATE]: Term 1 election: Requested vote from peers 04f5f856ae004e06bfe8d74881ec7371 (127.0.98.1:42393), caf1cf8c0a49436fb92743a2cddc285c (127.0.98.3:42925)
I20260926 01:10:00.720357 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "801368c3811a450a9f0aefacd9cd2f79" candidate_uuid: "f873f7c4a5944baab02c621fee691375" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04f5f856ae004e06bfe8d74881ec7371"
I20260926 01:10:00.720371 1942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "801368c3811a450a9f0aefacd9cd2f79" candidate_uuid: "f873f7c4a5944baab02c621fee691375" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "caf1cf8c0a49436fb92743a2cddc285c"
I20260926 01:10:00.720427 1661 raft_consensus.cc:3037] T 801368c3811a450a9f0aefacd9cd2f79 P 04f5f856ae004e06bfe8d74881ec7371 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:10:00.720451 1942 raft_consensus.cc:3037] T 801368c3811a450a9f0aefacd9cd2f79 P caf1cf8c0a49436fb92743a2cddc285c [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:10:00.721076 1661 raft_consensus.cc:2454] T 801368c3811a450a9f0aefacd9cd2f79 P 04f5f856ae004e06bfe8d74881ec7371 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f873f7c4a5944baab02c621fee691375 in term 1.
I20260926 01:10:00.721131 1942 raft_consensus.cc:2454] T 801368c3811a450a9f0aefacd9cd2f79 P caf1cf8c0a49436fb92743a2cddc285c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f873f7c4a5944baab02c621fee691375 in term 1.
I20260926 01:10:00.721264 1732 leader_election.cc:302] T 801368c3811a450a9f0aefacd9cd2f79 P f873f7c4a5944baab02c621fee691375 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: caf1cf8c0a49436fb92743a2cddc285c, f873f7c4a5944baab02c621fee691375; no voters:
I20260926 01:10:00.721355 2019 raft_consensus.cc:2781] T 801368c3811a450a9f0aefacd9cd2f79 P f873f7c4a5944baab02c621fee691375 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:10:00.721488 2019 raft_consensus.cc:686] T 801368c3811a450a9f0aefacd9cd2f79 P f873f7c4a5944baab02c621fee691375 [term 1 LEADER]: Becoming Leader. State: Replica: f873f7c4a5944baab02c621fee691375, State: Running, Role: LEADER
I20260926 01:10:00.721560 2019 consensus_queue.cc:237] T 801368c3811a450a9f0aefacd9cd2f79 P f873f7c4a5944baab02c621fee691375 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04f5f856ae004e06bfe8d74881ec7371" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42393 } } peers { permanent_uuid: "f873f7c4a5944baab02c621fee691375" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37623 } } peers { permanent_uuid: "caf1cf8c0a49436fb92743a2cddc285c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42925 } }
I20260926 01:10:00.722313 1504 catalog_manager.cc:5495] T 801368c3811a450a9f0aefacd9cd2f79 P f873f7c4a5944baab02c621fee691375 reported cstate change: term changed from 0 to 1, leader changed from <none> to f873f7c4a5944baab02c621fee691375 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "f873f7c4a5944baab02c621fee691375" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04f5f856ae004e06bfe8d74881ec7371" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42393 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f873f7c4a5944baab02c621fee691375" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37623 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "caf1cf8c0a49436fb92743a2cddc285c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42925 } health_report { overall_health: UNKNOWN } } }
W20260926 01:10:00.768682 1989 tablet.cc:2369] T 801368c3811a450a9f0aefacd9cd2f79 P caf1cf8c0a49436fb92743a2cddc285c: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 26 01:10:00 dist-test-slave-7b22 krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384999, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:10:00.781170 1880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:00.774682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59142 (local address 127.0.98.62:42075)
0926 01:10:00.774744 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:00.774746 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:00.774800 (+ 54us) server_negotiation.cc:407] Connection header received
0926 01:10:00.774856 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:00.774856 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:00.774864 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:00.774911 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:00.775330 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:00.775850 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:00.776514 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:00.776735 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:00.776786 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:00.778830 (+ 2044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:10:00.778832 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:10:00.778832 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:10:00.778838 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:10:00.780134 (+ 1296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:00.780485 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:00.780486 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:00.780486 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:00.780533 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:00.780714 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:00.780714 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:00.780715 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:00.780816 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:10:00.780864 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:00.781013 (+ 149us) server_negotiation.cc:299] Negotiation successful
0926 01:10:00.781084 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 01:10:00.781328 2049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:00.774549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42075 (local address 127.0.0.1:59142)
0926 01:10:00.774732 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:00.774742 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:00.774812 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:00.774924 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:00.774925 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:00.775228 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:10:00.775276 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:00.775277 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:00.775913 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:00.775913 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:00.776444 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:00.776446 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:00.776791 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:00.776792 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:00.776828 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:00.777303 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:10:00.777308 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:10:00.778715 (+ 1407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:10:00.780214 (+ 1499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:10:00.780221 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:10:00.780224 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:10:00.780428 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:10:00.780597 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:10:00.780598 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:10:00.780598 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:10:00.780666 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:10:00.780873 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:10:00.780879 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:10:00.780934 (+ 55us) client_negotiation.cc:769] Sending connection context
0926 01:10:00.780993 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 01:10:00.781216 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"mutex_wait_us":85,"thread_start_us":86,"threads_started":1}
I20260926 01:10:00.783404 1504 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59142:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 01:10:00.783622 1504 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:10:00.785082 1781 tablet_service.cc:1467] Processing CreateTablet for tablet f98818ab8d524809bbb59178fe2077a8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=bf94e84cd9c4454d82b6510c938d5919]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 01:10:00.785082 1641 tablet_service.cc:1467] Processing CreateTablet for tablet f98818ab8d524809bbb59178fe2077a8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=bf94e84cd9c4454d82b6510c938d5919]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 01:10:00.785246 1641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f98818ab8d524809bbb59178fe2077a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:10:00.785246 1781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f98818ab8d524809bbb59178fe2077a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:10:00.786751 2013 tablet_bootstrap.cc:492] T f98818ab8d524809bbb59178fe2077a8 P f873f7c4a5944baab02c621fee691375: Bootstrap starting.
I20260926 01:10:00.786844 1922 tablet_service.cc:1467] Processing CreateTablet for tablet f98818ab8d524809bbb59178fe2077a8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=bf94e84cd9c4454d82b6510c938d5919]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 01:10:00.786975 1922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f98818ab8d524809bbb59178fe2077a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:10:00.787164 2013 tablet_bootstrap.cc:654] T f98818ab8d524809bbb59178fe2077a8 P f873f7c4a5944baab02c621fee691375: Neither blocks nor log segments found. Creating new log.
I20260926 01:10:00.787781 2013 tablet_bootstrap.cc:492] T f98818ab8d524809bbb59178fe2077a8 P f873f7c4a5944baab02c621fee691375: No bootstrap required, opened a new log
I20260926 01:10:00.787827 2013 ts_tablet_manager.cc:1397] T f98818ab8d524809bbb59178fe2077a8 P f873f7c4a5944baab02c621fee691375: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260926 01:10:00.788081 2011 tablet_bootstrap.cc:492] T f98818ab8d524809bbb59178fe2077a8 P caf1cf8c0a49436fb92743a2cddc285c: Bootstrap starting.
I20260926 01:10:00.788107 2014 tablet_bootstrap.cc:492] T f98818ab8d524809bbb59178fe2077a8 P 04f5f856ae004e06bfe8d74881ec7371: Bootstrap starting.
I20260926 01:10:00.788401 2014 tablet_bootstrap.cc:654] T f98818ab8d524809bbb59178fe2077a8 P 04f5f856ae004e06bfe8d74881ec7371: Neither blocks nor log segments found. Creating new log.
I20260926 01:10:00.788441 2011 tablet_bootstrap.cc:654] T f98818ab8d524809bbb59178fe2077a8 P caf1cf8c0a49436fb92743a2cddc285c: Neither blocks nor log segments found. Creating new log.
I20260926 01:10:00.788489 2013 raft_consensus.cc:348] T f98818ab8d524809bbb59178fe2077a8 P f873f7c4a5944baab02c621fee691375 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04f5f856ae004e06bfe8d74881ec7371" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42393 } } peers { permanent_uuid: "f873f7c4a5944baab02c621fee691375" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37623 } } peers { permanent_uuid: "caf1cf8c0a49436fb92743a2cddc285c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42925 } }
I20260926 01:10:00.788571 2013 raft_consensus.cc:374] T f98818ab8d524809bbb59178fe2077a8 P f873f7c4a5944baab02c621fee691375 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:10:00.788587 2013 raft_consensus.cc:729] T f98818ab8d524809bbb59178fe2077a8 P f873f7c4a5944baab02c621fee691375 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f873f7c4a5944baab02c621fee691375, State: Initialized, Role: FOLLOWER
I20260926 01:10:00.788635 2013 consensus_queue.cc:260] T f98818ab8d524809bbb59178fe2077a8 P f873f7c4a5944baab02c621fee691375 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04f5f856ae004e06bfe8d74881ec7371" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42393 } } peers { permanent_uuid: "f873f7c4a5944baab02c621fee691375" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37623 } } peers { permanent_uuid: "caf1cf8c0a49436fb92743a2cddc285c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42925 } }
I20260926 01:10:00.788731 2013 ts_tablet_manager.cc:1428] T f98818ab8d524809bbb59178fe2077a8 P f873f7c4a5944baab02c621fee691375: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260926 01:10:00.788869 2019 tablet_replica.cc:439] T f98818ab8d524809bbb59178fe2077a8 P f873f7c4a5944baab02c621fee691375: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04f5f856ae004e06bfe8d74881ec7371" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42393 } } peers { permanent_uuid: "f873f7c4a5944baab02c621fee691375" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37623 } } peers { permanent_uuid: "caf1cf8c0a49436fb92743a2cddc285c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42925 } } }
I20260926 01:10:00.788924 2019 tablet_replica.cc:442] T f98818ab8d524809bbb59178fe2077a8 P f873f7c4a5944baab02c621fee691375: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 01:10:00.788939 2014 tablet_bootstrap.cc:492] T f98818ab8d524809bbb59178fe2077a8 P 04f5f856ae004e06bfe8d74881ec7371: No bootstrap required, opened a new log
I20260926 01:10:00.788971 2014 ts_tablet_manager.cc:1397] T f98818ab8d524809bbb59178fe2077a8 P 04f5f856ae004e06bfe8d74881ec7371: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 01:10:00.789045 2014 raft_consensus.cc:348] T f98818ab8d524809bbb59178fe2077a8 P 04f5f856ae004e06bfe8d74881ec7371 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04f5f856ae004e06bfe8d74881ec7371" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42393 } } peers { permanent_uuid: "f873f7c4a5944baab02c621fee691375" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37623 } } peers { permanent_uuid: "caf1cf8c0a49436fb92743a2cddc285c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42925 } }
I20260926 01:10:00.789115 2014 raft_consensus.cc:374] T f98818ab8d524809bbb59178fe2077a8 P 04f5f856ae004e06bfe8d74881ec7371 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:10:00.789142 2014 raft_consensus.cc:729] T f98818ab8d524809bbb59178fe2077a8 P 04f5f856ae004e06bfe8d74881ec7371 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04f5f856ae004e06bfe8d74881ec7371, State: Initialized, Role: FOLLOWER
I20260926 01:10:00.789180 2014 consensus_queue.cc:260] T f98818ab8d524809bbb59178fe2077a8 P 04f5f856ae004e06bfe8d74881ec7371 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04f5f856ae004e06bfe8d74881ec7371" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42393 } } peers { permanent_uuid: "f873f7c4a5944baab02c621fee691375" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37623 } } peers { permanent_uuid: "caf1cf8c0a49436fb92743a2cddc285c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42925 } }
I20260926 01:10:00.789263 2014 ts_tablet_manager.cc:1428] T f98818ab8d524809bbb59178fe2077a8 P 04f5f856ae004e06bfe8d74881ec7371: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:00.789345 2022 tablet_replica.cc:439] T f98818ab8d524809bbb59178fe2077a8 P 04f5f856ae004e06bfe8d74881ec7371: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04f5f856ae004e06bfe8d74881ec7371" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42393 } } peers { permanent_uuid: "f873f7c4a5944baab02c621fee691375" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37623 } } peers { permanent_uuid: "caf1cf8c0a49436fb92743a2cddc285c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42925 } } }
I20260926 01:10:00.789405 2022 tablet_replica.cc:442] T f98818ab8d524809bbb59178fe2077a8 P 04f5f856ae004e06bfe8d74881ec7371: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 01:10:00.789640 2011 tablet_bootstrap.cc:492] T f98818ab8d524809bbb59178fe2077a8 P caf1cf8c0a49436fb92743a2cddc285c: No bootstrap required, opened a new log
I20260926 01:10:00.789702 2011 ts_tablet_manager.cc:1397] T f98818ab8d524809bbb59178fe2077a8 P caf1cf8c0a49436fb92743a2cddc285c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 01:10:00.789889 2011 raft_consensus.cc:348] T f98818ab8d524809bbb59178fe2077a8 P caf1cf8c0a49436fb92743a2cddc285c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04f5f856ae004e06bfe8d74881ec7371" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42393 } } peers { permanent_uuid: "f873f7c4a5944baab02c621fee691375" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37623 } } peers { permanent_uuid: "caf1cf8c0a49436fb92743a2cddc285c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42925 } }
I20260926 01:10:00.789959 2011 raft_consensus.cc:374] T f98818ab8d524809bbb59178fe2077a8 P caf1cf8c0a49436fb92743a2cddc285c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:10:00.789983 2011 raft_consensus.cc:729] T f98818ab8d524809bbb59178fe2077a8 P caf1cf8c0a49436fb92743a2cddc285c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: caf1cf8c0a49436fb92743a2cddc285c, State: Initialized, Role: FOLLOWER
I20260926 01:10:00.790025 2011 consensus_queue.cc:260] T f98818ab8d524809bbb59178fe2077a8 P caf1cf8c0a49436fb92743a2cddc285c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04f5f856ae004e06bfe8d74881ec7371" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42393 } } peers { permanent_uuid: "f873f7c4a5944baab02c621fee691375" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37623 } } peers { permanent_uuid: "caf1cf8c0a49436fb92743a2cddc285c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42925 } }
I20260926 01:10:00.790143 2011 ts_tablet_manager.cc:1428] T f98818ab8d524809bbb59178fe2077a8 P caf1cf8c0a49436fb92743a2cddc285c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:00.790179 2015 tablet_replica.cc:439] T f98818ab8d524809bbb59178fe2077a8 P caf1cf8c0a49436fb92743a2cddc285c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04f5f856ae004e06bfe8d74881ec7371" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42393 } } peers { permanent_uuid: "f873f7c4a5944baab02c621fee691375" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37623 } } peers { permanent_uuid: "caf1cf8c0a49436fb92743a2cddc285c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42925 } } }
I20260926 01:10:00.790266 2015 tablet_replica.cc:442] T f98818ab8d524809bbb59178fe2077a8 P caf1cf8c0a49436fb92743a2cddc285c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 01:10:00.896337 2019 raft_consensus.cc:482] T f98818ab8d524809bbb59178fe2077a8 P f873f7c4a5944baab02c621fee691375 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:10:00.896440 2019 raft_consensus.cc:504] T f98818ab8d524809bbb59178fe2077a8 P f873f7c4a5944baab02c621fee691375 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04f5f856ae004e06bfe8d74881ec7371" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42393 } } peers { permanent_uuid: "f873f7c4a5944baab02c621fee691375" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37623 } } peers { permanent_uuid: "caf1cf8c0a49436fb92743a2cddc285c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42925 } }
I20260926 01:10:00.896580 2019 leader_election.cc:288] T f98818ab8d524809bbb59178fe2077a8 P f873f7c4a5944baab02c621fee691375 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 04f5f856ae004e06bfe8d74881ec7371 (127.0.98.1:42393), caf1cf8c0a49436fb92743a2cddc285c (127.0.98.3:42925)
I20260926 01:10:00.896828 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f98818ab8d524809bbb59178fe2077a8" candidate_uuid: "f873f7c4a5944baab02c621fee691375" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04f5f856ae004e06bfe8d74881ec7371" is_pre_election: true
I20260926 01:10:00.896914 1661 raft_consensus.cc:2454] T f98818ab8d524809bbb59178fe2077a8 P 04f5f856ae004e06bfe8d74881ec7371 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f873f7c4a5944baab02c621fee691375 in term 0.
I20260926 01:10:00.896907 1942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f98818ab8d524809bbb59178fe2077a8" candidate_uuid: "f873f7c4a5944baab02c621fee691375" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "caf1cf8c0a49436fb92743a2cddc285c" is_pre_election: true
I20260926 01:10:00.897019 1942 raft_consensus.cc:2454] T f98818ab8d524809bbb59178fe2077a8 P caf1cf8c0a49436fb92743a2cddc285c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f873f7c4a5944baab02c621fee691375 in term 0.
I20260926 01:10:00.897063 1732 leader_election.cc:302] T f98818ab8d524809bbb59178fe2077a8 P f873f7c4a5944baab02c621fee691375 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04f5f856ae004e06bfe8d74881ec7371, f873f7c4a5944baab02c621fee691375; no voters:
I20260926 01:10:00.897157 2019 raft_consensus.cc:2781] T f98818ab8d524809bbb59178fe2077a8 P f873f7c4a5944baab02c621fee691375 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:10:00.897202 2019 raft_consensus.cc:482] T f98818ab8d524809bbb59178fe2077a8 P f873f7c4a5944baab02c621fee691375 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:10:00.897217 2019 raft_consensus.cc:3037] T f98818ab8d524809bbb59178fe2077a8 P f873f7c4a5944baab02c621fee691375 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:10:00.897981 2019 raft_consensus.cc:504] T f98818ab8d524809bbb59178fe2077a8 P f873f7c4a5944baab02c621fee691375 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04f5f856ae004e06bfe8d74881ec7371" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42393 } } peers { permanent_uuid: "f873f7c4a5944baab02c621fee691375" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37623 } } peers { permanent_uuid: "caf1cf8c0a49436fb92743a2cddc285c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42925 } }
I20260926 01:10:00.898094 2019 leader_election.cc:288] T f98818ab8d524809bbb59178fe2077a8 P f873f7c4a5944baab02c621fee691375 [CANDIDATE]: Term 1 election: Requested vote from peers 04f5f856ae004e06bfe8d74881ec7371 (127.0.98.1:42393), caf1cf8c0a49436fb92743a2cddc285c (127.0.98.3:42925)
I20260926 01:10:00.898283 1942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f98818ab8d524809bbb59178fe2077a8" candidate_uuid: "f873f7c4a5944baab02c621fee691375" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "caf1cf8c0a49436fb92743a2cddc285c"
I20260926 01:10:00.898355 1942 raft_consensus.cc:3037] T f98818ab8d524809bbb59178fe2077a8 P caf1cf8c0a49436fb92743a2cddc285c [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:10:00.898360 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f98818ab8d524809bbb59178fe2077a8" candidate_uuid: "f873f7c4a5944baab02c621fee691375" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04f5f856ae004e06bfe8d74881ec7371"
I20260926 01:10:00.898435 1661 raft_consensus.cc:3037] T f98818ab8d524809bbb59178fe2077a8 P 04f5f856ae004e06bfe8d74881ec7371 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:10:00.899097 1661 raft_consensus.cc:2454] T f98818ab8d524809bbb59178fe2077a8 P 04f5f856ae004e06bfe8d74881ec7371 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f873f7c4a5944baab02c621fee691375 in term 1.
I20260926 01:10:00.899113 1942 raft_consensus.cc:2454] T f98818ab8d524809bbb59178fe2077a8 P caf1cf8c0a49436fb92743a2cddc285c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f873f7c4a5944baab02c621fee691375 in term 1.
I20260926 01:10:00.899245 1732 leader_election.cc:302] T f98818ab8d524809bbb59178fe2077a8 P f873f7c4a5944baab02c621fee691375 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04f5f856ae004e06bfe8d74881ec7371, f873f7c4a5944baab02c621fee691375; no voters:
I20260926 01:10:00.899366 2019 raft_consensus.cc:2781] T f98818ab8d524809bbb59178fe2077a8 P f873f7c4a5944baab02c621fee691375 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:10:00.899418 2019 raft_consensus.cc:686] T f98818ab8d524809bbb59178fe2077a8 P f873f7c4a5944baab02c621fee691375 [term 1 LEADER]: Becoming Leader. State: Replica: f873f7c4a5944baab02c621fee691375, State: Running, Role: LEADER
I20260926 01:10:00.899466 2019 consensus_queue.cc:237] T f98818ab8d524809bbb59178fe2077a8 P f873f7c4a5944baab02c621fee691375 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04f5f856ae004e06bfe8d74881ec7371" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42393 } } peers { permanent_uuid: "f873f7c4a5944baab02c621fee691375" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37623 } } peers { permanent_uuid: "caf1cf8c0a49436fb92743a2cddc285c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42925 } }
I20260926 01:10:00.899791 2042 tablet_replica.cc:439] T f98818ab8d524809bbb59178fe2077a8 P f873f7c4a5944baab02c621fee691375: TxnStatusTablet state changed. Reason: New leader f873f7c4a5944baab02c621fee691375. Latest consensus state: current_term: 1 leader_uuid: "f873f7c4a5944baab02c621fee691375" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04f5f856ae004e06bfe8d74881ec7371" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42393 } } peers { permanent_uuid: "f873f7c4a5944baab02c621fee691375" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37623 } } peers { permanent_uuid: "caf1cf8c0a49436fb92743a2cddc285c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42925 } } }
I20260926 01:10:00.899873 2042 tablet_replica.cc:442] T f98818ab8d524809bbb59178fe2077a8 P f873f7c4a5944baab02c621fee691375: This TxnStatusTablet replica's current role is: LEADER
I20260926 01:10:00.900033 2054 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 01:10:00.900152 2054 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 01:10:00.900146 1503 catalog_manager.cc:5495] T f98818ab8d524809bbb59178fe2077a8 P f873f7c4a5944baab02c621fee691375 reported cstate change: term changed from 0 to 1, leader changed from <none> to f873f7c4a5944baab02c621fee691375 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "f873f7c4a5944baab02c621fee691375" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04f5f856ae004e06bfe8d74881ec7371" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42393 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f873f7c4a5944baab02c621fee691375" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37623 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "caf1cf8c0a49436fb92743a2cddc285c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42925 } health_report { overall_health: UNKNOWN } } }
I20260926 01:10:00.906173 2049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:00.903164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37623 (local address 127.0.0.1:52606)
0926 01:10:00.903245 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:00.903257 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:00.903300 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:00.903512 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:00.903513 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:00.903515 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 01:10:00.903610 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:00.903612 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:00.904513 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:00.904514 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:00.905152 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:00.905154 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:00.905634 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:00.905634 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:00.905680 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:00.905696 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 01:10:00.906030 (+ 334us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 01:10:00.906034 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 01:10:00.906066 (+ 32us) client_negotiation.cc:240] Negotiation successful
0926 01:10:00.906112 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260926 01:10:00.906188 2007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:00.903206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52606 (local address 127.0.98.2:37623)
0926 01:10:00.903384 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:00.903386 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:00.903393 (+ 7us) server_negotiation.cc:407] Connection header received
0926 01:10:00.903420 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:00.903421 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:00.903438 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:00.903499 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:10:00.903681 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:00.904427 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:00.905233 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:00.905548 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:00.905617 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:00.905783 (+ 166us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:10:00.905921 (+ 138us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:10:00.906049 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:00.906087 (+ 38us) server_negotiation.cc:299] Negotiation successful
0926 01:10:00.906110 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156}
I20260926 01:10:00.907246 1942 raft_consensus.cc:1261] T f98818ab8d524809bbb59178fe2077a8 P caf1cf8c0a49436fb92743a2cddc285c [term 1 FOLLOWER]: Refusing update from remote peer f873f7c4a5944baab02c621fee691375: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:10:00.907361 1661 raft_consensus.cc:1261] T f98818ab8d524809bbb59178fe2077a8 P 04f5f856ae004e06bfe8d74881ec7371 [term 1 FOLLOWER]: Refusing update from remote peer f873f7c4a5944baab02c621fee691375: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:10:00.907624 2019 consensus_queue.cc:1035] T f98818ab8d524809bbb59178fe2077a8 P f873f7c4a5944baab02c621fee691375 [LEADER]: Connected to new peer: Peer: permanent_uuid: "caf1cf8c0a49436fb92743a2cddc285c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:10:00.907773 2042 consensus_queue.cc:1035] T f98818ab8d524809bbb59178fe2077a8 P f873f7c4a5944baab02c621fee691375 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04f5f856ae004e06bfe8d74881ec7371" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:10:00.908424 2022 tablet_replica.cc:439] T f98818ab8d524809bbb59178fe2077a8 P 04f5f856ae004e06bfe8d74881ec7371: TxnStatusTablet state changed. Reason: New leader f873f7c4a5944baab02c621fee691375. Latest consensus state: current_term: 1 leader_uuid: "f873f7c4a5944baab02c621fee691375" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04f5f856ae004e06bfe8d74881ec7371" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42393 } } peers { permanent_uuid: "f873f7c4a5944baab02c621fee691375" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37623 } } peers { permanent_uuid: "caf1cf8c0a49436fb92743a2cddc285c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42925 } } }
I20260926 01:10:00.908440 2015 tablet_replica.cc:439] T f98818ab8d524809bbb59178fe2077a8 P caf1cf8c0a49436fb92743a2cddc285c: TxnStatusTablet state changed. Reason: New leader f873f7c4a5944baab02c621fee691375. Latest consensus state: current_term: 1 leader_uuid: "f873f7c4a5944baab02c621fee691375" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04f5f856ae004e06bfe8d74881ec7371" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42393 } } peers { permanent_uuid: "f873f7c4a5944baab02c621fee691375" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37623 } } peers { permanent_uuid: "caf1cf8c0a49436fb92743a2cddc285c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42925 } } }
I20260926 01:10:00.908517 2022 tablet_replica.cc:442] T f98818ab8d524809bbb59178fe2077a8 P 04f5f856ae004e06bfe8d74881ec7371: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 01:10:00.908524 2015 tablet_replica.cc:442] T f98818ab8d524809bbb59178fe2077a8 P caf1cf8c0a49436fb92743a2cddc285c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 01:10:00.909121 2030 mvcc.cc:205] Tried to move back new op lower bound from 7333416963714265088 to 7333416963684798464. Current Snapshot: MvccSnapshot[applied={T|T < 7333416963714265088}]
I20260926 01:10:00.909304 2042 tablet_replica.cc:439] T f98818ab8d524809bbb59178fe2077a8 P f873f7c4a5944baab02c621fee691375: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f873f7c4a5944baab02c621fee691375" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04f5f856ae004e06bfe8d74881ec7371" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42393 } } peers { permanent_uuid: "f873f7c4a5944baab02c621fee691375" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37623 } } peers { permanent_uuid: "caf1cf8c0a49436fb92743a2cddc285c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42925 } } }
I20260926 01:10:00.909397 2042 tablet_replica.cc:442] T f98818ab8d524809bbb59178fe2077a8 P f873f7c4a5944baab02c621fee691375: This TxnStatusTablet replica's current role is: LEADER
I20260926 01:10:00.909632 2015 tablet_replica.cc:439] T f98818ab8d524809bbb59178fe2077a8 P caf1cf8c0a49436fb92743a2cddc285c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f873f7c4a5944baab02c621fee691375" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04f5f856ae004e06bfe8d74881ec7371" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42393 } } peers { permanent_uuid: "f873f7c4a5944baab02c621fee691375" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37623 } } peers { permanent_uuid: "caf1cf8c0a49436fb92743a2cddc285c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42925 } } }
I20260926 01:10:00.909704 2015 tablet_replica.cc:442] T f98818ab8d524809bbb59178fe2077a8 P caf1cf8c0a49436fb92743a2cddc285c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 01:10:00.909858 2019 tablet_replica.cc:439] T f98818ab8d524809bbb59178fe2077a8 P f873f7c4a5944baab02c621fee691375: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f873f7c4a5944baab02c621fee691375" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04f5f856ae004e06bfe8d74881ec7371" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42393 } } peers { permanent_uuid: "f873f7c4a5944baab02c621fee691375" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37623 } } peers { permanent_uuid: "caf1cf8c0a49436fb92743a2cddc285c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42925 } } }
I20260926 01:10:00.909929 2019 tablet_replica.cc:442] T f98818ab8d524809bbb59178fe2077a8 P f873f7c4a5944baab02c621fee691375: This TxnStatusTablet replica's current role is: LEADER
I20260926 01:10:00.910717 2022 tablet_replica.cc:439] T f98818ab8d524809bbb59178fe2077a8 P 04f5f856ae004e06bfe8d74881ec7371: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f873f7c4a5944baab02c621fee691375" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04f5f856ae004e06bfe8d74881ec7371" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42393 } } peers { permanent_uuid: "f873f7c4a5944baab02c621fee691375" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37623 } } peers { permanent_uuid: "caf1cf8c0a49436fb92743a2cddc285c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42925 } } }
I20260926 01:10:00.910806 2022 tablet_replica.cc:442] T f98818ab8d524809bbb59178fe2077a8 P 04f5f856ae004e06bfe8d74881ec7371: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 01:10:00.914247 2007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:00.911632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52618 (local address 127.0.98.2:37623)
0926 01:10:00.911695 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:00.911697 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:00.911704 (+ 7us) server_negotiation.cc:407] Connection header received
0926 01:10:00.911721 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:00.911721 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:00.911730 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:00.911797 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:10:00.911930 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:00.912463 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:00.913569 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:00.913857 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:00.913914 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:00.914062 (+ 148us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:10:00.914090 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:10:00.914124 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:00.914175 (+ 51us) server_negotiation.cc:299] Negotiation successful
0926 01:10:00.914192 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 01:10:00.919503 2007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:00.916695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45809 (local address 127.0.98.2:37623)
0926 01:10:00.916733 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:00.916734 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:00.916812 (+ 78us) server_negotiation.cc:407] Connection header received
0926 01:10:00.916878 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:00.916879 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:00.916885 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:00.916925 (+ 40us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:10:00.917057 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:00.917539 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:00.918706 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:00.918883 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:00.918939 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:00.919227 (+ 288us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:10:00.919257 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:10:00.919307 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:00.919390 (+ 83us) server_negotiation.cc:299] Negotiation successful
0926 01:10:00.919410 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 01:10:00.919589 2061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:00.916631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37623 (local address 127.0.98.2:45809)
0926 01:10:00.916760 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:00.916770 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:00.916826 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:00.916930 (+ 104us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:00.916931 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:00.916932 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 01:10:00.916997 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:00.916998 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:00.917601 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:00.917602 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:00.918627 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:00.918630 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:00.918983 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:00.918984 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:00.919089 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:00.919110 (+ 21us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 01:10:00.919326 (+ 216us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 01:10:00.919328 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 01:10:00.919392 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 01:10:00.919436 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":68,"threads_started":1}
I20260926 01:10:00.921633 1661 raft_consensus.cc:1261] T 801368c3811a450a9f0aefacd9cd2f79 P 04f5f856ae004e06bfe8d74881ec7371 [term 1 FOLLOWER]: Refusing update from remote peer f873f7c4a5944baab02c621fee691375: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:10:00.921633 1942 raft_consensus.cc:1261] T 801368c3811a450a9f0aefacd9cd2f79 P caf1cf8c0a49436fb92743a2cddc285c [term 1 FOLLOWER]: Refusing update from remote peer f873f7c4a5944baab02c621fee691375: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:10:00.921866 2019 consensus_queue.cc:1035] T 801368c3811a450a9f0aefacd9cd2f79 P f873f7c4a5944baab02c621fee691375 [LEADER]: Connected to new peer: Peer: permanent_uuid: "caf1cf8c0a49436fb92743a2cddc285c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:10:00.921969 2042 consensus_queue.cc:1035] T 801368c3811a450a9f0aefacd9cd2f79 P f873f7c4a5944baab02c621fee691375 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04f5f856ae004e06bfe8d74881ec7371" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:10:00.923985 2026 mvcc.cc:205] Tried to move back new op lower bound from 7333416963773755392 to 7333416962955939840. Current Snapshot: MvccSnapshot[applied={T|T < 7333416963773755392}]
I20260926 01:10:00.936470 1503 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38136:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:10:00.936573 1503 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38136:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:10:00.937196 1503 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a3fc3ff0f3ad4ec19518afe126ee308f: Sending DeleteTablet for 3 replicas of tablet 801368c3811a450a9f0aefacd9cd2f79
I20260926 01:10:00.937445 1641 tablet_service.cc:1514] Processing DeleteTablet for tablet 801368c3811a450a9f0aefacd9cd2f79 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:10:00 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59704
I20260926 01:10:00.937495 1922 tablet_service.cc:1514] Processing DeleteTablet for tablet 801368c3811a450a9f0aefacd9cd2f79 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:10:00 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56922
I20260926 01:10:00.937565 2034 tablet_replica.cc:330] T 801368c3811a450a9f0aefacd9cd2f79 P 04f5f856ae004e06bfe8d74881ec7371: stopping tablet replica
I20260926 01:10:00.937613 2034 raft_consensus.cc:2229] T 801368c3811a450a9f0aefacd9cd2f79 P 04f5f856ae004e06bfe8d74881ec7371 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:10:00.937650 2034 raft_consensus.cc:2258] T 801368c3811a450a9f0aefacd9cd2f79 P 04f5f856ae004e06bfe8d74881ec7371 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:10:00.937621 1780 tablet_service.cc:1514] Processing DeleteTablet for tablet 801368c3811a450a9f0aefacd9cd2f79 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:10:00 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53726
I20260926 01:10:00.937727 2035 tablet_replica.cc:330] T 801368c3811a450a9f0aefacd9cd2f79 P caf1cf8c0a49436fb92743a2cddc285c: stopping tablet replica
I20260926 01:10:00.937775 2035 raft_consensus.cc:2229] T 801368c3811a450a9f0aefacd9cd2f79 P caf1cf8c0a49436fb92743a2cddc285c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:10:00.937784 2036 tablet_replica.cc:330] T 801368c3811a450a9f0aefacd9cd2f79 P f873f7c4a5944baab02c621fee691375: stopping tablet replica
I20260926 01:10:00.937810 2035 raft_consensus.cc:2258] T 801368c3811a450a9f0aefacd9cd2f79 P caf1cf8c0a49436fb92743a2cddc285c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:10:00.937852 2036 raft_consensus.cc:2229] T 801368c3811a450a9f0aefacd9cd2f79 P f873f7c4a5944baab02c621fee691375 [term 1 LEADER]: Raft consensus shutting down.
I20260926 01:10:00.937945 2036 raft_consensus.cc:2258] T 801368c3811a450a9f0aefacd9cd2f79 P f873f7c4a5944baab02c621fee691375 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:10:00.938186 2035 ts_tablet_manager.cc:1905] T 801368c3811a450a9f0aefacd9cd2f79 P caf1cf8c0a49436fb92743a2cddc285c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:10:00.938375 2036 ts_tablet_manager.cc:1905] T 801368c3811a450a9f0aefacd9cd2f79 P f873f7c4a5944baab02c621fee691375: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:10:00.939941 2034 ts_tablet_manager.cc:1905] T 801368c3811a450a9f0aefacd9cd2f79 P 04f5f856ae004e06bfe8d74881ec7371: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:10:00.940148 2035 ts_tablet_manager.cc:1918] T 801368c3811a450a9f0aefacd9cd2f79 P caf1cf8c0a49436fb92743a2cddc285c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 01:10:00.940191 2035 log.cc:1196] T 801368c3811a450a9f0aefacd9cd2f79 P caf1cf8c0a49436fb92743a2cddc285c: Deleting WAL directory at /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384993574762-392-0/minicluster-data/ts-2/wal/wals/801368c3811a450a9f0aefacd9cd2f79
I20260926 01:10:00.940339 2035 ts_tablet_manager.cc:1939] T 801368c3811a450a9f0aefacd9cd2f79 P caf1cf8c0a49436fb92743a2cddc285c: Deleting consensus metadata
I20260926 01:10:00.940630 1488 catalog_manager.cc:4841] TS caf1cf8c0a49436fb92743a2cddc285c (127.0.98.3:42925): tablet 801368c3811a450a9f0aefacd9cd2f79 (table test-table [id=3f17b7aee917453387e105b9a01e1a79]) successfully deleted
I20260926 01:10:00.941497 2034 ts_tablet_manager.cc:1918] T 801368c3811a450a9f0aefacd9cd2f79 P 04f5f856ae004e06bfe8d74881ec7371: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 01:10:00.941545 2034 log.cc:1196] T 801368c3811a450a9f0aefacd9cd2f79 P 04f5f856ae004e06bfe8d74881ec7371: Deleting WAL directory at /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384993574762-392-0/minicluster-data/ts-0/wal/wals/801368c3811a450a9f0aefacd9cd2f79
I20260926 01:10:00.941725 2034 ts_tablet_manager.cc:1939] T 801368c3811a450a9f0aefacd9cd2f79 P 04f5f856ae004e06bfe8d74881ec7371: Deleting consensus metadata
I20260926 01:10:00.941895 2036 ts_tablet_manager.cc:1918] T 801368c3811a450a9f0aefacd9cd2f79 P f873f7c4a5944baab02c621fee691375: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 01:10:00.941939 2036 log.cc:1196] T 801368c3811a450a9f0aefacd9cd2f79 P f873f7c4a5944baab02c621fee691375: Deleting WAL directory at /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384993574762-392-0/minicluster-data/ts-1/wal/wals/801368c3811a450a9f0aefacd9cd2f79
I20260926 01:10:00.942148 2036 ts_tablet_manager.cc:1939] T 801368c3811a450a9f0aefacd9cd2f79 P f873f7c4a5944baab02c621fee691375: Deleting consensus metadata
I20260926 01:10:00.942442 1489 catalog_manager.cc:4841] TS f873f7c4a5944baab02c621fee691375 (127.0.98.2:37623): tablet 801368c3811a450a9f0aefacd9cd2f79 (table test-table [id=3f17b7aee917453387e105b9a01e1a79]) successfully deleted
I20260926 01:10:00.942667 1488 catalog_manager.cc:4841] TS 04f5f856ae004e06bfe8d74881ec7371 (127.0.98.1:42393): tablet 801368c3811a450a9f0aefacd9cd2f79 (table test-table [id=3f17b7aee917453387e105b9a01e1a79]) successfully deleted
Sep 26 01:10:00 dist-test-slave-7b22 krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385000, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 01:10:00.948939 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:00.940024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54738 (local address 127.0.98.1:42393)
0926 01:10:00.940068 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:00.940070 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:00.940646 (+ 576us) server_negotiation.cc:407] Connection header received
0926 01:10:00.940747 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:00.940748 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:00.940758 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:00.940805 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:00.941302 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:00.941771 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:00.942748 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:00.942973 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:00.943030 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:00.946053 (+ 3023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:10:00.946062 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:10:00.946063 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:10:00.946070 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:10:00.947804 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:00.948129 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:00.948132 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:00.948133 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:00.948195 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:00.948351 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:00.948352 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:00.948352 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:00.948523 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:10:00.948660 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:00.948749 (+ 89us) server_negotiation.cc:299] Negotiation successful
0926 01:10:00.948831 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
W20260926 01:10:00.949198 1701 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54738
I20260926 01:10:00.956342 1880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:00.950210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59152 (local address 127.0.98.62:42075)
0926 01:10:00.950256 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:00.950258 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:00.950314 (+ 56us) server_negotiation.cc:407] Connection header received
0926 01:10:00.950377 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:00.950378 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:00.950388 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:00.950435 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:00.950906 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:00.951391 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:00.952047 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:00.952246 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:00.952301 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:00.953442 (+ 1141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:10:00.953444 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:10:00.953444 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:10:00.953451 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:10:00.955090 (+ 1639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:00.955429 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:00.955430 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:00.955431 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:00.955500 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:00.955727 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:00.955728 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:00.955728 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:00.955877 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:10:00.955971 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:00.956162 (+ 191us) server_negotiation.cc:299] Negotiation successful
0926 01:10:00.956240 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
W20260926 01:10:00.956588 1503 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59152
I20260926 01:10:00.957269 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 1576
I20260926 01:10:00.962839 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 1716
I20260926 01:10:00.967329 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 1856
I20260926 01:10:00.971257 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 1472
2026-09-26T01:10:00Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2795 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384993574762-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 01:10:01 dist-test-slave-7b22 krb5kdc[2072](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 01:10:01 dist-test-slave-7b22 krb5kdc[2072](info): set up 2 sockets
Sep 26 01:10:01 dist-test-slave-7b22 krb5kdc[2072](info): commencing operation
krb5kdc: starting...
W20260926 01:10:01.929351 392 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384993574762-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384993574762-392-0/krb5kdc/test-user.keytab.
Sep 26 01:10:02 dist-test-slave-7b22 krb5kdc[2072](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385002, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T01:10:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:10:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384993574762-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384993574762-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 01:10:02.044628 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384993574762-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384993574762-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384993574762-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43913
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384993574762-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43913
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:10:02.125608 2085 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:10:02.125862 2085 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:10:02.125895 2085 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:10:02.127068 2085 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:10:02.127111 2085 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 01:10:02.127171 2085 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:10:02.127202 2085 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:10:02.127228 2085 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:10:02.127250 2085 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:10:02.128789 2085 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384993574762-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384993574762-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43913
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384993574762-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43913
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2085
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384993574762-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:10:02.129035 2085 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:10:02.129251 2085 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:10:02.133874 2091 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:10:02.133956 2090 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:10:02.133921 2093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:10:02.134006 2085 server_base.cc:1034] running on GCE node
I20260926 01:10:02.134358 2085 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:10:02.134644 2085 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:10:02.135792 2085 hybrid_clock.cc:648] HybridClock initialized: now 1790385002135778 us; error 30 us; skew 500 ppm
Sep 26 01:10:02 dist-test-slave-7b22 krb5kdc[2072](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385002, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:10:02.138450 2085 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 01:10:02.138871 2085 webserver.cc:460] Webserver started at http://127.0.98.62:38227/ using document root <none> and password file <none>
I20260926 01:10:02.139076 2085 fs_manager.cc:359] Metadata directory not provided
I20260926 01:10:02.139122 2085 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:10:02.139251 2085 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:10:02.140306 2085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384993574762-392-0/minicluster-data/master-0/data/instance:
uuid: "43da41c6d9f34b998e1c66111660121f"
format_stamp: "Formatted at 2026-09-26 01:10:02 on dist-test-slave-7b22"
server_key: "2443341be120b71bba43048a4981ab9e"
server_key_iv: "4291c517585d585c2f4b0e6e4a721ec8"
server_key_version: "encryptionkey@0"
I20260926 01:10:02.140633 2085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384993574762-392-0/minicluster-data/master-0/wal/instance:
uuid: "43da41c6d9f34b998e1c66111660121f"
format_stamp: "Formatted at 2026-09-26 01:10:02 on dist-test-slave-7b22"
server_key: "2443341be120b71bba43048a4981ab9e"
server_key_iv: "4291c517585d585c2f4b0e6e4a721ec8"
server_key_version: "encryptionkey@0"
I20260926 01:10:02.141961 2085 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 01:10:02.142765 2100 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:02.143105 2085 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:10:02.143218 2085 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384993574762-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384993574762-392-0/minicluster-data/master-0/wal
uuid: "43da41c6d9f34b998e1c66111660121f"
format_stamp: "Formatted at 2026-09-26 01:10:02 on dist-test-slave-7b22"
server_key: "2443341be120b71bba43048a4981ab9e"
server_key_iv: "4291c517585d585c2f4b0e6e4a721ec8"
server_key_version: "encryptionkey@0"
I20260926 01:10:02.143292 2085 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384993574762-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384993574762-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384993574762-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:10:02.149916 2085 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:10:02.151870 2085 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:10:02.152009 2085 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:10:02.156325 2085 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43913
I20260926 01:10:02.156389 2162 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43913 every 8 connection(s)
I20260926 01:10:02.156690 2085 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
I20260926 01:10:02.157352 2163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:10:02.159261 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 2085
I20260926 01:10:02.159343 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384993574762-392-0/minicluster-data/master-0/wal/instance
I20260926 01:10:02.159490 392 external_mini_cluster.cc:1428] Setting key 0e691e31cb0a9d3190692ea063ab81b4
I20260926 01:10:02.159700 2163 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43da41c6d9f34b998e1c66111660121f: Bootstrap starting.
I20260926 01:10:02.160419 2163 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 43da41c6d9f34b998e1c66111660121f: Neither blocks nor log segments found. Creating new log.
I20260926 01:10:02.160733 2163 log.cc:824] T 00000000000000000000000000000000 P 43da41c6d9f34b998e1c66111660121f: Log is configured to *not* fsync() on all Append() calls
I20260926 01:10:02.161626 2163 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43da41c6d9f34b998e1c66111660121f: No bootstrap required, opened a new log
I20260926 01:10:02.164117 2163 raft_consensus.cc:348] T 00000000000000000000000000000000 P 43da41c6d9f34b998e1c66111660121f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43da41c6d9f34b998e1c66111660121f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43913 } }
I20260926 01:10:02.164302 2163 raft_consensus.cc:374] T 00000000000000000000000000000000 P 43da41c6d9f34b998e1c66111660121f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:10:02.164336 2163 raft_consensus.cc:729] T 00000000000000000000000000000000 P 43da41c6d9f34b998e1c66111660121f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43da41c6d9f34b998e1c66111660121f, State: Initialized, Role: FOLLOWER
I20260926 01:10:02.164461 2163 consensus_queue.cc:260] T 00000000000000000000000000000000 P 43da41c6d9f34b998e1c66111660121f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43da41c6d9f34b998e1c66111660121f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43913 } }
I20260926 01:10:02.164533 2163 raft_consensus.cc:388] T 00000000000000000000000000000000 P 43da41c6d9f34b998e1c66111660121f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:10:02.164562 2163 raft_consensus.cc:482] T 00000000000000000000000000000000 P 43da41c6d9f34b998e1c66111660121f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:10:02.164599 2163 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 43da41c6d9f34b998e1c66111660121f [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:10:02.165251 2163 raft_consensus.cc:504] T 00000000000000000000000000000000 P 43da41c6d9f34b998e1c66111660121f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43da41c6d9f34b998e1c66111660121f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43913 } }
I20260926 01:10:02.165337 2163 leader_election.cc:302] T 00000000000000000000000000000000 P 43da41c6d9f34b998e1c66111660121f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 43da41c6d9f34b998e1c66111660121f; no voters:
I20260926 01:10:02.165463 2163 leader_election.cc:288] T 00000000000000000000000000000000 P 43da41c6d9f34b998e1c66111660121f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:10:02.165516 2168 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 43da41c6d9f34b998e1c66111660121f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:10:02.165617 2168 raft_consensus.cc:686] T 00000000000000000000000000000000 P 43da41c6d9f34b998e1c66111660121f [term 1 LEADER]: Becoming Leader. State: Replica: 43da41c6d9f34b998e1c66111660121f, State: Running, Role: LEADER
I20260926 01:10:02.165688 2168 consensus_queue.cc:237] T 00000000000000000000000000000000 P 43da41c6d9f34b998e1c66111660121f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43da41c6d9f34b998e1c66111660121f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43913 } }
I20260926 01:10:02.165723 2163 sys_catalog.cc:564] T 00000000000000000000000000000000 P 43da41c6d9f34b998e1c66111660121f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:10:02.166102 2168 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43da41c6d9f34b998e1c66111660121f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 43da41c6d9f34b998e1c66111660121f. Latest consensus state: current_term: 1 leader_uuid: "43da41c6d9f34b998e1c66111660121f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43da41c6d9f34b998e1c66111660121f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43913 } } }
I20260926 01:10:02.166114 2169 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43da41c6d9f34b998e1c66111660121f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "43da41c6d9f34b998e1c66111660121f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43da41c6d9f34b998e1c66111660121f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43913 } } }
I20260926 01:10:02.166183 2168 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43da41c6d9f34b998e1c66111660121f [sys.catalog]: This master's current role is: LEADER
I20260926 01:10:02.166211 2169 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43da41c6d9f34b998e1c66111660121f [sys.catalog]: This master's current role is: LEADER
I20260926 01:10:02.166371 2175 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:10:02.166860 2175 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 26 01:10:02 dist-test-slave-7b22 krb5kdc[2072](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385002, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:10:02.169088 2175 catalog_manager.cc:1347] Generated new cluster ID: 4776c0383c00407583b96b0fd14c4248
I20260926 01:10:02.169147 2175 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 01:10:02.170951 2166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:02.159954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49702 (local address 127.0.98.62:43913)
0926 01:10:02.160131 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:02.160142 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:02.160176 (+ 34us) server_negotiation.cc:407] Connection header received
0926 01:10:02.160762 (+ 586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:02.160771 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:02.160863 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:02.161020 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:02.161763 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:02.163137 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:02.164224 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:02.164818 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:02.164928 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:02.167620 (+ 2692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:10:02.167624 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:10:02.167624 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:10:02.167636 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:10:02.169486 (+ 1850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:02.169857 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:02.169861 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:02.169861 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:02.169959 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:02.170138 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:02.170139 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:02.170140 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:02.170390 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:10:02.170555 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:02.170631 (+ 76us) server_negotiation.cc:299] Negotiation successful
0926 01:10:02.170768 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":77,"threads_started":1}
I20260926 01:10:02.177588 2175 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:10:02.178333 2175 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:10:02.181010 2175 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 43da41c6d9f34b998e1c66111660121f: Generated new TSK 0
I20260926 01:10:02.181193 2175 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384993574762-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384993574762-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 01:10:02.207031 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384993574762-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384993574762-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384993574762-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384993574762-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43913
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384993574762-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:10:02.288033 2189 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:10:02.288266 2189 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:10:02.288292 2189 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:10:02.289428 2189 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:10:02.289471 2189 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:10:02.289491 2189 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 01:10:02.289501 2189 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 01:10:02.289517 2189 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 01:10:02.291005 2189 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384993574762-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384993574762-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384993574762-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384993574762-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:43913
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2189
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384993574762-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:10:02.291270 2189 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:10:02.291486 2189 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:10:02.296262 2197 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:10:02.296262 2194 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:10:02.296291 2195 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:10:02.296499 2189 server_base.cc:1034] running on GCE node
I20260926 01:10:02.296687 2189 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:10:02.296888 2189 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:10:02.298028 2189 hybrid_clock.cc:648] HybridClock initialized: now 1790385002298014 us; error 30 us; skew 500 ppm
Sep 26 01:10:02 dist-test-slave-7b22 krb5kdc[2072](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385002, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:10:02.300601 2189 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 01:10:02.301052 2189 webserver.cc:460] Webserver started at http://127.0.98.1:39561/ using document root <none> and password file <none>
I20260926 01:10:02.301259 2189 fs_manager.cc:359] Metadata directory not provided
I20260926 01:10:02.301303 2189 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:10:02.301419 2189 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:10:02.302522 2189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384993574762-392-0/minicluster-data/ts-0/data/instance:
uuid: "61c33320fe9844018afb635f9dee3641"
format_stamp: "Formatted at 2026-09-26 01:10:02 on dist-test-slave-7b22"
server_key: "2e8d0175f17eae0ddc78465949daaa5c"
server_key_iv: "a6d08c2f74ffe9d55d75208d03335596"
server_key_version: "encryptionkey@0"
I20260926 01:10:02.302845 2189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384993574762-392-0/minicluster-data/ts-0/wal/instance:
uuid: "61c33320fe9844018afb635f9dee3641"
format_stamp: "Formatted at 2026-09-26 01:10:02 on dist-test-slave-7b22"
server_key: "2e8d0175f17eae0ddc78465949daaa5c"
server_key_iv: "a6d08c2f74ffe9d55d75208d03335596"
server_key_version: "encryptionkey@0"
I20260926 01:10:02.304114 2189 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 01:10:02.304788 2204 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:02.304921 2189 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 01:10:02.305002 2189 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384993574762-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384993574762-392-0/minicluster-data/ts-0/wal
uuid: "61c33320fe9844018afb635f9dee3641"
format_stamp: "Formatted at 2026-09-26 01:10:02 on dist-test-slave-7b22"
server_key: "2e8d0175f17eae0ddc78465949daaa5c"
server_key_iv: "a6d08c2f74ffe9d55d75208d03335596"
server_key_version: "encryptionkey@0"
I20260926 01:10:02.305066 2189 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384993574762-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384993574762-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384993574762-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:10:02.318392 2189 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:10:02.320199 2189 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:10:02.320389 2189 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:10:02.320888 2189 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:10:02.320932 2189 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:02.320960 2189 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:10:02.320978 2189 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:02.327171 2189 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35639
I20260926 01:10:02.327231 2319 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35639 every 8 connection(s)
I20260926 01:10:02.327550 2189 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384993574762-392-0/minicluster-data/ts-0/data/info.pb
Sep 26 01:10:02 dist-test-slave-7b22 krb5kdc[2072](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385002, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:10:02.330972 2166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:02.321355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52537 (local address 127.0.98.62:43913)
0926 01:10:02.321400 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:02.321402 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:02.321626 (+ 224us) server_negotiation.cc:407] Connection header received
0926 01:10:02.322202 (+ 576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:02.322204 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:02.322217 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:02.322281 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:02.323088 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:02.323877 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:02.325386 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:02.325626 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:02.325677 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:02.328293 (+ 2616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:10:02.328295 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:10:02.328295 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:10:02.328303 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:10:02.329577 (+ 1274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:02.330021 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:02.330022 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:02.330022 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:02.330076 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:02.330342 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:02.330342 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:02.330342 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:02.330465 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:10:02.330510 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:02.330834 (+ 324us) server_negotiation.cc:299] Negotiation successful
0926 01:10:02.330905 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 01:10:02.331149 2213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:02.321350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43913 (local address 127.0.98.1:52537)
0926 01:10:02.321512 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:02.321546 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:02.322109 (+ 563us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:02.322358 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:02.322363 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:02.322698 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:10:02.323002 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:02.323009 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:02.323981 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:02.323982 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:02.325284 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:02.325287 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:02.325709 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:02.325710 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:02.325881 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:02.326477 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:10:02.326489 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:10:02.328195 (+ 1706us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:10:02.329655 (+ 1460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:10:02.329662 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:10:02.329664 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:10:02.329947 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:10:02.330150 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:10:02.330151 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:10:02.330151 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:10:02.330271 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:10:02.330539 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:10:02.330546 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:10:02.330719 (+ 173us) client_negotiation.cc:769] Sending connection context
0926 01:10:02.330813 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 01:10:02.330948 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":91,"threads_started":1}
I20260926 01:10:02.331696 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 2189
I20260926 01:10:02.331769 2320 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43913
I20260926 01:10:02.331808 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384993574762-392-0/minicluster-data/ts-0/wal/instance
I20260926 01:10:02.331892 2320 heartbeater.cc:464] Registering TS with master...
I20260926 01:10:02.331986 392 external_mini_cluster.cc:1428] Setting key 04a72b5fdb548427f6526c7363f08076
I20260926 01:10:02.332084 2320 heartbeater.cc:511] Master 127.0.98.62:43913 requested a full tablet report, sending...
I20260926 01:10:02.332932 2117 ts_manager.cc:194] Registered new tserver with Master: 61c33320fe9844018afb635f9dee3641 (127.0.98.1:35639)
I20260926 01:10:02.333724 2117 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52537
I20260926 01:10:02.338521 2166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:02.332815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43079 (local address 127.0.98.62:43913)
0926 01:10:02.332851 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:02.332853 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:02.332963 (+ 110us) server_negotiation.cc:407] Connection header received
0926 01:10:02.333017 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:02.333018 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:02.333027 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:02.333072 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:02.333359 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:02.333872 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:02.334849 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:02.335034 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:02.335090 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:02.336035 (+ 945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:10:02.336036 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:10:02.336036 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:10:02.336042 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:10:02.337457 (+ 1415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:02.337865 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:02.337866 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:02.337866 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:02.337917 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:02.338091 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:02.338092 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:02.338092 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:02.338187 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:10:02.338232 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:02.338398 (+ 166us) server_negotiation.cc:299] Negotiation successful
0926 01:10:02.338456 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 01:10:02.338608 2326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:02.332756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43913 (local address 127.0.98.1:43079)
0926 01:10:02.332902 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:02.332911 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:02.332965 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:02.333076 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:02.333077 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:02.333239 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:10:02.333297 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:02.333299 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:02.333974 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:02.333975 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:02.334750 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:02.334753 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:02.335102 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:02.335103 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:02.335167 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:02.335533 (+ 366us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:10:02.335538 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:10:02.335937 (+ 399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:10:02.337539 (+ 1602us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:10:02.337540 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:10:02.337541 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:10:02.337763 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:10:02.337971 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:10:02.337972 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:10:02.337972 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:10:02.338037 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:10:02.338236 (+ 199us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:10:02.338237 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:10:02.338286 (+ 49us) client_negotiation.cc:769] Sending connection context
0926 01:10:02.338380 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 01:10:02.338489 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":73,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384993574762-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384993574762-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 01:10:02.356757 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384993574762-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384993574762-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384993574762-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384993574762-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43913
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384993574762-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:10:02.437562 2329 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:10:02.437774 2329 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:10:02.437812 2329 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:10:02.439018 2329 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:10:02.439070 2329 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:10:02.439100 2329 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 01:10:02.439116 2329 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 01:10:02.439148 2329 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 01:10:02.440647 2329 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384993574762-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384993574762-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384993574762-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384993574762-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:43913
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2329
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384993574762-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:10:02.440884 2329 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:10:02.441108 2329 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:10:02.445899 2334 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:10:02.445899 2337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:10:02.445904 2335 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:10:02.446166 2329 server_base.cc:1034] running on GCE node
I20260926 01:10:02.446359 2329 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:10:02.446625 2329 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:10:02.447778 2329 hybrid_clock.cc:648] HybridClock initialized: now 1790385002447763 us; error 29 us; skew 500 ppm
Sep 26 01:10:02 dist-test-slave-7b22 krb5kdc[2072](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385002, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:10:02.450383 2329 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 01:10:02.450773 2329 webserver.cc:460] Webserver started at http://127.0.98.2:44021/ using document root <none> and password file <none>
I20260926 01:10:02.450994 2329 fs_manager.cc:359] Metadata directory not provided
I20260926 01:10:02.451040 2329 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:10:02.451136 2329 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:10:02.452144 2329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384993574762-392-0/minicluster-data/ts-1/data/instance:
uuid: "166bbea04fd64826a5aae830ef8bfbae"
format_stamp: "Formatted at 2026-09-26 01:10:02 on dist-test-slave-7b22"
server_key: "10788f7cb270c6a9cefe70a84014b4f9"
server_key_iv: "77df00ad6a4946ca4588f3f252ad2f0c"
server_key_version: "encryptionkey@0"
I20260926 01:10:02.452450 2329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384993574762-392-0/minicluster-data/ts-1/wal/instance:
uuid: "166bbea04fd64826a5aae830ef8bfbae"
format_stamp: "Formatted at 2026-09-26 01:10:02 on dist-test-slave-7b22"
server_key: "10788f7cb270c6a9cefe70a84014b4f9"
server_key_iv: "77df00ad6a4946ca4588f3f252ad2f0c"
server_key_version: "encryptionkey@0"
I20260926 01:10:02.453660 2329 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 01:10:02.454402 2344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:02.454591 2329 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 01:10:02.454679 2329 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384993574762-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384993574762-392-0/minicluster-data/ts-1/wal
uuid: "166bbea04fd64826a5aae830ef8bfbae"
format_stamp: "Formatted at 2026-09-26 01:10:02 on dist-test-slave-7b22"
server_key: "10788f7cb270c6a9cefe70a84014b4f9"
server_key_iv: "77df00ad6a4946ca4588f3f252ad2f0c"
server_key_version: "encryptionkey@0"
I20260926 01:10:02.454746 2329 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384993574762-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384993574762-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384993574762-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:10:02.483103 2329 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:10:02.485312 2329 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:10:02.485438 2329 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:10:02.485980 2329 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:10:02.486024 2329 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:02.486052 2329 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:10:02.486066 2329 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:02.492393 2329 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43727
I20260926 01:10:02.492506 2459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43727 every 8 connection(s)
I20260926 01:10:02.492838 2329 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384993574762-392-0/minicluster-data/ts-1/data/info.pb
Sep 26 01:10:02 dist-test-slave-7b22 krb5kdc[2072](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385002, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:10:02.496441 2166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:02.486409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36567 (local address 127.0.98.62:43913)
0926 01:10:02.486478 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:02.486481 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:02.486689 (+ 208us) server_negotiation.cc:407] Connection header received
0926 01:10:02.487216 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:02.487217 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:02.487230 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:02.487281 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:02.488147 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:02.488934 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:02.490540 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:02.490837 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:02.490930 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:02.493794 (+ 2864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:10:02.493796 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:10:02.493796 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:10:02.493804 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:10:02.495077 (+ 1273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:02.495480 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:02.495481 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:02.495481 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:02.495543 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:02.495796 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:02.495796 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:02.495797 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:02.495924 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:10:02.495988 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:02.496258 (+ 270us) server_negotiation.cc:299] Negotiation successful
0926 01:10:02.496342 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 01:10:02.496656 2353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:02.486394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43913 (local address 127.0.98.2:36567)
0926 01:10:02.486570 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:02.486610 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:02.487110 (+ 500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:02.487356 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:02.487360 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:02.487721 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:10:02.488025 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:02.488031 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:02.489050 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:02.489052 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:02.490444 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:02.490448 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:02.490948 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:02.490949 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:02.491098 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:02.491801 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:10:02.491812 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:10:02.493699 (+ 1887us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:10:02.495168 (+ 1469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:10:02.495175 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:10:02.495178 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:10:02.495404 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:10:02.495611 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:10:02.495612 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:10:02.495612 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:10:02.495725 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:10:02.495984 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:10:02.495990 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:10:02.496135 (+ 145us) client_negotiation.cc:769] Sending connection context
0926 01:10:02.496232 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 01:10:02.496412 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":106,"threads_started":1}
I20260926 01:10:02.498961 2460 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43913
I20260926 01:10:02.499056 2460 heartbeater.cc:464] Registering TS with master...
I20260926 01:10:02.499277 2460 heartbeater.cc:511] Master 127.0.98.62:43913 requested a full tablet report, sending...
I20260926 01:10:02.499706 2116 ts_manager.cc:194] Registered new tserver with Master: 166bbea04fd64826a5aae830ef8bfbae (127.0.98.2:43727)
I20260926 01:10:02.500200 2116 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:36567
I20260926 01:10:02.501955 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 2329
I20260926 01:10:02.502044 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384993574762-392-0/minicluster-data/ts-1/wal/instance
I20260926 01:10:02.502224 392 external_mini_cluster.cc:1428] Setting key 3a52a556985aec83e4d45a826a3e9ed3
I20260926 01:10:02.505502 2166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:02.498356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60935 (local address 127.0.98.62:43913)
0926 01:10:02.498422 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:02.498424 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:02.498520 (+ 96us) server_negotiation.cc:407] Connection header received
0926 01:10:02.498584 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:02.498585 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:02.498596 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:02.498653 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:02.499039 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:02.499715 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:02.500748 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:02.501048 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:02.501093 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:02.502332 (+ 1239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:10:02.502333 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:10:02.502333 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:10:02.502340 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:10:02.503988 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:02.504562 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:02.504563 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:02.504563 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:02.504645 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:02.504859 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:02.504860 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:02.504861 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:02.505074 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:10:02.505125 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:02.505371 (+ 246us) server_negotiation.cc:299] Negotiation successful
0926 01:10:02.505439 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 01:10:02.505509 2466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:02.498291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43913 (local address 127.0.98.2:60935)
0926 01:10:02.498457 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:02.498467 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:02.498525 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:02.498658 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:02.498659 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:02.498847 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:10:02.498925 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:02.498927 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:02.499849 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:02.499850 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:02.500635 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:02.500639 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:02.501180 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:02.501180 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:02.501246 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:02.501738 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:10:02.501743 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:10:02.502238 (+ 495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:10:02.504091 (+ 1853us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:10:02.504092 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:10:02.504092 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:10:02.504436 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:10:02.504724 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:10:02.504725 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:10:02.504725 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:10:02.504789 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:10:02.505177 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:10:02.505178 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:10:02.505230 (+ 52us) client_negotiation.cc:769] Sending connection context
0926 01:10:02.505312 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 01:10:02.505420 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":80,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384993574762-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384993574762-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 01:10:02.528301 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384993574762-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384993574762-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384993574762-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384993574762-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43913
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384993574762-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:10:02.610188 2469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:10:02.610388 2469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:10:02.610414 2469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:10:02.611568 2469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:10:02.611610 2469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:10:02.611629 2469 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 01:10:02.611640 2469 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 01:10:02.611668 2469 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 01:10:02.613135 2469 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384993574762-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384993574762-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384993574762-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384993574762-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:43913
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2469
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384993574762-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:10:02.613330 2469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:10:02.613516 2469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:10:02.618165 2475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:10:02.618216 2474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:10:02.618165 2477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:10:02.618394 2469 server_base.cc:1034] running on GCE node
I20260926 01:10:02.618584 2469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:10:02.618810 2469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:10:02.619942 2469 hybrid_clock.cc:648] HybridClock initialized: now 1790385002619927 us; error 34 us; skew 500 ppm
Sep 26 01:10:02 dist-test-slave-7b22 krb5kdc[2072](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385002, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:10:02.622428 2469 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 01:10:02.622843 2469 webserver.cc:460] Webserver started at http://127.0.98.3:46145/ using document root <none> and password file <none>
I20260926 01:10:02.623036 2469 fs_manager.cc:359] Metadata directory not provided
I20260926 01:10:02.623075 2469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:10:02.623158 2469 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:10:02.624135 2469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384993574762-392-0/minicluster-data/ts-2/data/instance:
uuid: "d73081e8b2f449138126fc26b5f3a5c2"
format_stamp: "Formatted at 2026-09-26 01:10:02 on dist-test-slave-7b22"
server_key: "2a19b7b17a4969ea4185fe46e94e89df"
server_key_iv: "84c0c13173a1738cb76dba77da7b1f73"
server_key_version: "encryptionkey@0"
I20260926 01:10:02.624440 2469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384993574762-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d73081e8b2f449138126fc26b5f3a5c2"
format_stamp: "Formatted at 2026-09-26 01:10:02 on dist-test-slave-7b22"
server_key: "2a19b7b17a4969ea4185fe46e94e89df"
server_key_iv: "84c0c13173a1738cb76dba77da7b1f73"
server_key_version: "encryptionkey@0"
I20260926 01:10:02.625623 2469 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 01:10:02.626386 2484 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:02.626561 2469 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 01:10:02.626652 2469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384993574762-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384993574762-392-0/minicluster-data/ts-2/wal
uuid: "d73081e8b2f449138126fc26b5f3a5c2"
format_stamp: "Formatted at 2026-09-26 01:10:02 on dist-test-slave-7b22"
server_key: "2a19b7b17a4969ea4185fe46e94e89df"
server_key_iv: "84c0c13173a1738cb76dba77da7b1f73"
server_key_version: "encryptionkey@0"
I20260926 01:10:02.626729 2469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384993574762-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384993574762-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384993574762-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:10:02.639449 2469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:10:02.641330 2469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:10:02.641459 2469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:10:02.642004 2469 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:10:02.642052 2469 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:02.642074 2469 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:10:02.642087 2469 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:02.648603 2469 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36989
I20260926 01:10:02.648949 2469 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384993574762-392-0/minicluster-data/ts-2/data/info.pb
Sep 26 01:10:02 dist-test-slave-7b22 krb5kdc[2072](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385002, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:10:02.649444 2599 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36989 every 8 connection(s)
I20260926 01:10:02.652045 2166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:02.642424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54925 (local address 127.0.98.62:43913)
0926 01:10:02.642488 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:02.642491 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:02.642763 (+ 272us) server_negotiation.cc:407] Connection header received
0926 01:10:02.643444 (+ 681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:02.643446 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:02.643456 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:02.643503 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:02.644368 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:02.645142 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:02.646766 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:02.647028 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:02.647089 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:02.649630 (+ 2541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:10:02.649631 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:10:02.649631 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:10:02.649638 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:10:02.650872 (+ 1234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:02.651269 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:02.651270 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:02.651270 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:02.651321 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:02.651572 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:02.651572 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:02.651572 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:02.651669 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:10:02.651713 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:02.651926 (+ 213us) server_negotiation.cc:299] Negotiation successful
0926 01:10:02.651983 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 01:10:02.652251 2493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:02.642397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43913 (local address 127.0.98.3:54925)
0926 01:10:02.642597 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:02.642668 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:02.643334 (+ 666us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:02.643595 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:02.643600 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:02.643960 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:10:02.644264 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:02.644272 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:02.645249 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:02.645251 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:02.646647 (+ 1396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:02.646650 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:02.647118 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:02.647119 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:02.647271 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:02.647863 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:10:02.647876 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:10:02.649550 (+ 1674us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:10:02.650950 (+ 1400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:10:02.650956 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:10:02.650958 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:10:02.651203 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:10:02.651377 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:10:02.651378 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:10:02.651378 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:10:02.651477 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:10:02.651712 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:10:02.651716 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:10:02.651836 (+ 120us) client_negotiation.cc:769] Sending connection context
0926 01:10:02.651915 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 01:10:02.652036 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":90,"threads_started":1}
I20260926 01:10:02.652751 2600 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43913
I20260926 01:10:02.652830 2600 heartbeater.cc:464] Registering TS with master...
I20260926 01:10:02.653024 2600 heartbeater.cc:511] Master 127.0.98.62:43913 requested a full tablet report, sending...
I20260926 01:10:02.653072 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 2469
I20260926 01:10:02.653165 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384993574762-392-0/minicluster-data/ts-2/wal/instance
I20260926 01:10:02.653367 2116 ts_manager.cc:194] Registered new tserver with Master: d73081e8b2f449138126fc26b5f3a5c2 (127.0.98.3:36989)
I20260926 01:10:02.653395 392 external_mini_cluster.cc:1428] Setting key 00339d9b506343c06bafd46cc364a3f5
I20260926 01:10:02.653749 2116 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:54925
I20260926 01:10:02.654371 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 01:10:02.659865 2166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:02.653903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60193 (local address 127.0.98.62:43913)
0926 01:10:02.653943 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:02.653945 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:02.654215 (+ 270us) server_negotiation.cc:407] Connection header received
0926 01:10:02.654266 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:02.654266 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:02.654276 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:02.654321 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:02.654663 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:02.655116 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:02.656023 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:02.656313 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:02.656382 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:02.657215 (+ 833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:10:02.657216 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:10:02.657217 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:10:02.657223 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:10:02.658774 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:02.659121 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:02.659122 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:02.659122 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:02.659177 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:02.659401 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:02.659402 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:02.659403 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:02.659525 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:10:02.659586 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:02.659739 (+ 153us) server_negotiation.cc:299] Negotiation successful
0926 01:10:02.659799 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 01:10:02.659929 2606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:02.654010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43913 (local address 127.0.98.3:60193)
0926 01:10:02.654150 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:02.654159 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:02.654212 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:02.654329 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:02.654330 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:02.654517 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:10:02.654587 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:02.654589 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:02.655211 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:02.655212 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:02.655921 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:02.655923 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:02.656380 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:02.656381 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:02.656437 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:02.656835 (+ 398us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:10:02.656838 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:10:02.657134 (+ 296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:10:02.658863 (+ 1729us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:10:02.658864 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:10:02.658864 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:10:02.659047 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:10:02.659251 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:10:02.659251 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:10:02.659252 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:10:02.659318 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:10:02.659590 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:10:02.659591 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:10:02.659638 (+ 47us) client_negotiation.cc:769] Sending connection context
0926 01:10:02.659718 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 01:10:02.659825 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":69,"threads_started":1}
Sep 26 01:10:02 dist-test-slave-7b22 krb5kdc[2072](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385002, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 01:10:02 dist-test-slave-7b22 krb5kdc[2072](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385002, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:10:02.676986 2166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:02.670517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49706 (local address 127.0.98.62:43913)
0926 01:10:02.670572 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:02.670574 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:02.670585 (+ 11us) server_negotiation.cc:407] Connection header received
0926 01:10:02.670695 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:02.670696 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:02.670705 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:02.670760 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:02.671280 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:02.671784 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:02.672455 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:02.672627 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:02.672669 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:02.674814 (+ 2145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:10:02.674815 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:10:02.674816 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:10:02.674823 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:10:02.676007 (+ 1184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:02.676316 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:02.676317 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:02.676317 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:02.676373 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:02.676519 (+ 146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:02.676520 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:02.676520 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:02.676672 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:10:02.676746 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:02.676857 (+ 111us) server_negotiation.cc:299] Negotiation successful
0926 01:10:02.676917 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 01:10:02.678514 2116 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49706:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:10:02.680124 2116 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:10:02.687054 2615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:02.682379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43727 (local address 127.0.0.1:38970)
0926 01:10:02.682541 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:02.682573 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:02.682648 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:02.683031 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:02.683034 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:02.683036 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:10:02.683210 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:02.683213 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:02.684960 (+ 1747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:02.684961 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:02.685876 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:02.685880 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:02.686777 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:02.686778 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:02.686848 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:02.686862 (+ 14us) client_negotiation.cc:769] Sending connection context
0926 01:10:02.686932 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 01:10:02.686976 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":60,"threads_started":1}
I20260926 01:10:02.687099 2618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:02.682707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38970 (local address 127.0.98.2:43727)
0926 01:10:02.682863 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:02.682868 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:02.682879 (+ 11us) server_negotiation.cc:407] Connection header received
0926 01:10:02.682921 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:02.682923 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:02.682939 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:02.683020 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:10:02.683296 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:02.684880 (+ 1584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:02.685965 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:02.686681 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:02.686758 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:02.686822 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:02.686957 (+ 135us) server_negotiation.cc:299] Negotiation successful
0926 01:10:02.686998 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":67,"threads_started":1}
I20260926 01:10:02.687525 2614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:02.682587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35639 (local address 127.0.0.1:53100)
0926 01:10:02.682936 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:02.682945 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:02.683006 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:02.683479 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:02.683480 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:02.683482 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:10:02.683517 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:02.683518 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:02.684517 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:02.684518 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:02.685856 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:02.685860 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:02.687285 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:02.687286 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:02.687348 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:02.687352 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 01:10:02.687399 (+ 47us) client_negotiation.cc:240] Negotiation successful
0926 01:10:02.687443 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"mutex_wait_us":33,"thread_start_us":32,"threads_started":1}
I20260926 01:10:02.687860 2619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:02.682711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53100 (local address 127.0.98.1:35639)
0926 01:10:02.682894 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:02.682897 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:02.683306 (+ 409us) server_negotiation.cc:407] Connection header received
0926 01:10:02.683353 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:02.683356 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:02.683371 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:02.683452 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:10:02.683573 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:02.684393 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:02.685937 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:02.686681 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:02.686758 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:02.686822 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:02.687686 (+ 864us) server_negotiation.cc:299] Negotiation successful
0926 01:10:02.687736 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":74,"threads_started":1}
I20260926 01:10:02.687850 2394 tablet_service.cc:1467] Processing CreateTablet for tablet ccc241eca243493a957eee8609bf8ad5 (DEFAULT_TABLE table=test-table [id=beabdfb270e849c38f09bbbe1de4a838]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:10:02.688041 2617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:02.682381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36989 (local address 127.0.0.1:42100)
0926 01:10:02.682739 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:02.682748 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:02.682783 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:02.683172 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:02.683173 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:02.683176 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:10:02.683220 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:02.683220 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:02.684366 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:02.684368 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:02.685861 (+ 1493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:02.685863 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:02.687830 (+ 1967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:02.687831 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:02.687894 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:02.687898 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 01:10:02.687939 (+ 41us) client_negotiation.cc:240] Negotiation successful
0926 01:10:02.687978 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"mutex_wait_us":48,"thread_start_us":69,"threads_started":1}
I20260926 01:10:02.688298 2394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccc241eca243493a957eee8609bf8ad5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:10:02.688340 2616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:02.682505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42100 (local address 127.0.98.3:36989)
0926 01:10:02.682897 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:02.682901 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:02.682908 (+ 7us) server_negotiation.cc:407] Connection header received
0926 01:10:02.682938 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:02.682940 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:02.682951 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:02.683015 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:10:02.683536 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:02.684266 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:02.686859 (+ 2593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:02.687638 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:02.687714 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:02.687771 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:02.688189 (+ 418us) server_negotiation.cc:299] Negotiation successful
0926 01:10:02.688229 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":94,"threads_started":1}
I20260926 01:10:02.688620 2254 tablet_service.cc:1467] Processing CreateTablet for tablet ccc241eca243493a957eee8609bf8ad5 (DEFAULT_TABLE table=test-table [id=beabdfb270e849c38f09bbbe1de4a838]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:10:02.688987 2254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccc241eca243493a957eee8609bf8ad5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:10:02.689059 2520 tablet_service.cc:1467] Processing CreateTablet for tablet ccc241eca243493a957eee8609bf8ad5 (DEFAULT_TABLE table=test-table [id=beabdfb270e849c38f09bbbe1de4a838]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:10:02.689399 2520 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccc241eca243493a957eee8609bf8ad5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:10:02.691711 2620 tablet_bootstrap.cc:492] T ccc241eca243493a957eee8609bf8ad5 P 166bbea04fd64826a5aae830ef8bfbae: Bootstrap starting.
I20260926 01:10:02.692427 2620 tablet_bootstrap.cc:654] T ccc241eca243493a957eee8609bf8ad5 P 166bbea04fd64826a5aae830ef8bfbae: Neither blocks nor log segments found. Creating new log.
I20260926 01:10:02.692828 2620 log.cc:824] T ccc241eca243493a957eee8609bf8ad5 P 166bbea04fd64826a5aae830ef8bfbae: Log is configured to *not* fsync() on all Append() calls
I20260926 01:10:02.692833 2621 tablet_bootstrap.cc:492] T ccc241eca243493a957eee8609bf8ad5 P d73081e8b2f449138126fc26b5f3a5c2: Bootstrap starting.
I20260926 01:10:02.693257 2622 tablet_bootstrap.cc:492] T ccc241eca243493a957eee8609bf8ad5 P 61c33320fe9844018afb635f9dee3641: Bootstrap starting.
I20260926 01:10:02.693753 2621 tablet_bootstrap.cc:654] T ccc241eca243493a957eee8609bf8ad5 P d73081e8b2f449138126fc26b5f3a5c2: Neither blocks nor log segments found. Creating new log.
I20260926 01:10:02.693866 2622 tablet_bootstrap.cc:654] T ccc241eca243493a957eee8609bf8ad5 P 61c33320fe9844018afb635f9dee3641: Neither blocks nor log segments found. Creating new log.
I20260926 01:10:02.693997 2620 tablet_bootstrap.cc:492] T ccc241eca243493a957eee8609bf8ad5 P 166bbea04fd64826a5aae830ef8bfbae: No bootstrap required, opened a new log
I20260926 01:10:02.694072 2620 ts_tablet_manager.cc:1397] T ccc241eca243493a957eee8609bf8ad5 P 166bbea04fd64826a5aae830ef8bfbae: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 01:10:02.694111 2621 log.cc:824] T ccc241eca243493a957eee8609bf8ad5 P d73081e8b2f449138126fc26b5f3a5c2: Log is configured to *not* fsync() on all Append() calls
I20260926 01:10:02.694196 2622 log.cc:824] T ccc241eca243493a957eee8609bf8ad5 P 61c33320fe9844018afb635f9dee3641: Log is configured to *not* fsync() on all Append() calls
I20260926 01:10:02.694836 2621 tablet_bootstrap.cc:492] T ccc241eca243493a957eee8609bf8ad5 P d73081e8b2f449138126fc26b5f3a5c2: No bootstrap required, opened a new log
I20260926 01:10:02.694907 2621 ts_tablet_manager.cc:1397] T ccc241eca243493a957eee8609bf8ad5 P d73081e8b2f449138126fc26b5f3a5c2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 01:10:02.696072 2620 raft_consensus.cc:348] T ccc241eca243493a957eee8609bf8ad5 P 166bbea04fd64826a5aae830ef8bfbae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "166bbea04fd64826a5aae830ef8bfbae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43727 } } peers { permanent_uuid: "d73081e8b2f449138126fc26b5f3a5c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36989 } } peers { permanent_uuid: "61c33320fe9844018afb635f9dee3641" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35639 } }
I20260926 01:10:02.696260 2620 raft_consensus.cc:374] T ccc241eca243493a957eee8609bf8ad5 P 166bbea04fd64826a5aae830ef8bfbae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:10:02.696295 2620 raft_consensus.cc:729] T ccc241eca243493a957eee8609bf8ad5 P 166bbea04fd64826a5aae830ef8bfbae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 166bbea04fd64826a5aae830ef8bfbae, State: Initialized, Role: FOLLOWER
I20260926 01:10:02.696439 2620 consensus_queue.cc:260] T ccc241eca243493a957eee8609bf8ad5 P 166bbea04fd64826a5aae830ef8bfbae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "166bbea04fd64826a5aae830ef8bfbae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43727 } } peers { permanent_uuid: "d73081e8b2f449138126fc26b5f3a5c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36989 } } peers { permanent_uuid: "61c33320fe9844018afb635f9dee3641" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35639 } }
I20260926 01:10:02.696725 2620 ts_tablet_manager.cc:1428] T ccc241eca243493a957eee8609bf8ad5 P 166bbea04fd64826a5aae830ef8bfbae: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 01:10:02.696863 2460 heartbeater.cc:503] Master 127.0.98.62:43913 was elected leader, sending a full tablet report...
I20260926 01:10:02.696818 2621 raft_consensus.cc:348] T ccc241eca243493a957eee8609bf8ad5 P d73081e8b2f449138126fc26b5f3a5c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "166bbea04fd64826a5aae830ef8bfbae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43727 } } peers { permanent_uuid: "d73081e8b2f449138126fc26b5f3a5c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36989 } } peers { permanent_uuid: "61c33320fe9844018afb635f9dee3641" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35639 } }
I20260926 01:10:02.696972 2621 raft_consensus.cc:374] T ccc241eca243493a957eee8609bf8ad5 P d73081e8b2f449138126fc26b5f3a5c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:10:02.697009 2621 raft_consensus.cc:729] T ccc241eca243493a957eee8609bf8ad5 P d73081e8b2f449138126fc26b5f3a5c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d73081e8b2f449138126fc26b5f3a5c2, State: Initialized, Role: FOLLOWER
I20260926 01:10:02.697116 2621 consensus_queue.cc:260] T ccc241eca243493a957eee8609bf8ad5 P d73081e8b2f449138126fc26b5f3a5c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "166bbea04fd64826a5aae830ef8bfbae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43727 } } peers { permanent_uuid: "d73081e8b2f449138126fc26b5f3a5c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36989 } } peers { permanent_uuid: "61c33320fe9844018afb635f9dee3641" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35639 } }
I20260926 01:10:02.697404 2621 ts_tablet_manager.cc:1428] T ccc241eca243493a957eee8609bf8ad5 P d73081e8b2f449138126fc26b5f3a5c2: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 01:10:02.697463 2600 heartbeater.cc:503] Master 127.0.98.62:43913 was elected leader, sending a full tablet report...
I20260926 01:10:02.698077 2622 tablet_bootstrap.cc:492] T ccc241eca243493a957eee8609bf8ad5 P 61c33320fe9844018afb635f9dee3641: No bootstrap required, opened a new log
I20260926 01:10:02.698205 2622 ts_tablet_manager.cc:1397] T ccc241eca243493a957eee8609bf8ad5 P 61c33320fe9844018afb635f9dee3641: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.001s
I20260926 01:10:02.699710 2622 raft_consensus.cc:348] T ccc241eca243493a957eee8609bf8ad5 P 61c33320fe9844018afb635f9dee3641 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "166bbea04fd64826a5aae830ef8bfbae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43727 } } peers { permanent_uuid: "d73081e8b2f449138126fc26b5f3a5c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36989 } } peers { permanent_uuid: "61c33320fe9844018afb635f9dee3641" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35639 } }
I20260926 01:10:02.699854 2622 raft_consensus.cc:374] T ccc241eca243493a957eee8609bf8ad5 P 61c33320fe9844018afb635f9dee3641 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:10:02.699875 2622 raft_consensus.cc:729] T ccc241eca243493a957eee8609bf8ad5 P 61c33320fe9844018afb635f9dee3641 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61c33320fe9844018afb635f9dee3641, State: Initialized, Role: FOLLOWER
I20260926 01:10:02.699963 2622 consensus_queue.cc:260] T ccc241eca243493a957eee8609bf8ad5 P 61c33320fe9844018afb635f9dee3641 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "166bbea04fd64826a5aae830ef8bfbae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43727 } } peers { permanent_uuid: "d73081e8b2f449138126fc26b5f3a5c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36989 } } peers { permanent_uuid: "61c33320fe9844018afb635f9dee3641" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35639 } }
I20260926 01:10:02.700202 2622 ts_tablet_manager.cc:1428] T ccc241eca243493a957eee8609bf8ad5 P 61c33320fe9844018afb635f9dee3641: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260926 01:10:02.700268 2320 heartbeater.cc:503] Master 127.0.98.62:43913 was elected leader, sending a full tablet report...
W20260926 01:10:02.743413 2461 tablet.cc:2369] T ccc241eca243493a957eee8609bf8ad5 P 166bbea04fd64826a5aae830ef8bfbae: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 01:10:02.762070 2626 raft_consensus.cc:482] T ccc241eca243493a957eee8609bf8ad5 P 166bbea04fd64826a5aae830ef8bfbae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:10:02.762210 2626 raft_consensus.cc:504] T ccc241eca243493a957eee8609bf8ad5 P 166bbea04fd64826a5aae830ef8bfbae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "166bbea04fd64826a5aae830ef8bfbae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43727 } } peers { permanent_uuid: "d73081e8b2f449138126fc26b5f3a5c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36989 } } peers { permanent_uuid: "61c33320fe9844018afb635f9dee3641" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35639 } }
I20260926 01:10:02.762527 2626 leader_election.cc:288] T ccc241eca243493a957eee8609bf8ad5 P 166bbea04fd64826a5aae830ef8bfbae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d73081e8b2f449138126fc26b5f3a5c2 (127.0.98.3:36989), 61c33320fe9844018afb635f9dee3641 (127.0.98.1:35639)
I20260926 01:10:02.766577 2616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:02.762867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53127 (local address 127.0.98.3:36989)
0926 01:10:02.762924 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:02.762926 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:02.762936 (+ 10us) server_negotiation.cc:407] Connection header received
0926 01:10:02.762965 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:02.762966 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:02.762979 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:02.763036 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:10:02.763291 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:02.764019 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:02.765560 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:02.766180 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:02.766258 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:02.766272 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:02.766458 (+ 186us) server_negotiation.cc:299] Negotiation successful
0926 01:10:02.766504 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 01:10:02.766577 2619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:02.762860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:32999 (local address 127.0.98.1:35639)
0926 01:10:02.762904 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:02.762905 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:02.763256 (+ 351us) server_negotiation.cc:407] Connection header received
0926 01:10:02.763294 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:02.763296 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:02.763308 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:02.763375 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:10:02.763559 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:02.764202 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:02.765560 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:02.766177 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:02.766278 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:02.766288 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:02.766459 (+ 171us) server_negotiation.cc:299] Negotiation successful
0926 01:10:02.766501 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 01:10:02.766667 2353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:02.762748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36989 (local address 127.0.98.2:53127)
0926 01:10:02.762810 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:02.762824 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:02.762884 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:02.763081 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:02.763082 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:02.763085 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:10:02.763163 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:02.763165 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:02.764135 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:02.764136 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:02.765445 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:02.765448 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:02.766311 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:02.766312 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:02.766378 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:02.766380 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 01:10:02.766437 (+ 57us) client_negotiation.cc:240] Negotiation successful
0926 01:10:02.766489 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":35}
I20260926 01:10:02.766678 2629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:02.762748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35639 (local address 127.0.98.2:32999)
0926 01:10:02.763091 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:02.763112 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:02.763182 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:02.763408 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:02.763409 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:02.763411 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:10:02.763488 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:02.763490 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:02.764280 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:02.764281 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:02.765445 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:02.765448 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:02.766287 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:02.766288 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:02.766370 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:02.766375 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 01:10:02.766437 (+ 62us) client_negotiation.cc:240] Negotiation successful
0926 01:10:02.766489 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"mutex_wait_us":107,"thread_start_us":123,"threads_started":1}
I20260926 01:10:02.767110 2554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccc241eca243493a957eee8609bf8ad5" candidate_uuid: "166bbea04fd64826a5aae830ef8bfbae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73081e8b2f449138126fc26b5f3a5c2" is_pre_election: true
I20260926 01:10:02.767117 2274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccc241eca243493a957eee8609bf8ad5" candidate_uuid: "166bbea04fd64826a5aae830ef8bfbae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61c33320fe9844018afb635f9dee3641" is_pre_election: true
I20260926 01:10:02.767256 2274 raft_consensus.cc:2454] T ccc241eca243493a957eee8609bf8ad5 P 61c33320fe9844018afb635f9dee3641 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 166bbea04fd64826a5aae830ef8bfbae in term 0.
I20260926 01:10:02.767263 2554 raft_consensus.cc:2454] T ccc241eca243493a957eee8609bf8ad5 P d73081e8b2f449138126fc26b5f3a5c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 166bbea04fd64826a5aae830ef8bfbae in term 0.
I20260926 01:10:02.767484 2348 leader_election.cc:302] T ccc241eca243493a957eee8609bf8ad5 P 166bbea04fd64826a5aae830ef8bfbae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 166bbea04fd64826a5aae830ef8bfbae, 61c33320fe9844018afb635f9dee3641; no voters:
I20260926 01:10:02.767659 2626 raft_consensus.cc:2781] T ccc241eca243493a957eee8609bf8ad5 P 166bbea04fd64826a5aae830ef8bfbae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:10:02.767735 2626 raft_consensus.cc:482] T ccc241eca243493a957eee8609bf8ad5 P 166bbea04fd64826a5aae830ef8bfbae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:10:02.767760 2626 raft_consensus.cc:3037] T ccc241eca243493a957eee8609bf8ad5 P 166bbea04fd64826a5aae830ef8bfbae [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:10:02.768751 2626 raft_consensus.cc:504] T ccc241eca243493a957eee8609bf8ad5 P 166bbea04fd64826a5aae830ef8bfbae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "166bbea04fd64826a5aae830ef8bfbae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43727 } } peers { permanent_uuid: "d73081e8b2f449138126fc26b5f3a5c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36989 } } peers { permanent_uuid: "61c33320fe9844018afb635f9dee3641" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35639 } }
I20260926 01:10:02.768882 2626 leader_election.cc:288] T ccc241eca243493a957eee8609bf8ad5 P 166bbea04fd64826a5aae830ef8bfbae [CANDIDATE]: Term 1 election: Requested vote from peers d73081e8b2f449138126fc26b5f3a5c2 (127.0.98.3:36989), 61c33320fe9844018afb635f9dee3641 (127.0.98.1:35639)
I20260926 01:10:02.769089 2274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccc241eca243493a957eee8609bf8ad5" candidate_uuid: "166bbea04fd64826a5aae830ef8bfbae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61c33320fe9844018afb635f9dee3641"
I20260926 01:10:02.769083 2554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccc241eca243493a957eee8609bf8ad5" candidate_uuid: "166bbea04fd64826a5aae830ef8bfbae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73081e8b2f449138126fc26b5f3a5c2"
I20260926 01:10:02.769179 2554 raft_consensus.cc:3037] T ccc241eca243493a957eee8609bf8ad5 P d73081e8b2f449138126fc26b5f3a5c2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:10:02.769181 2274 raft_consensus.cc:3037] T ccc241eca243493a957eee8609bf8ad5 P 61c33320fe9844018afb635f9dee3641 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:10:02.769892 2554 raft_consensus.cc:2454] T ccc241eca243493a957eee8609bf8ad5 P d73081e8b2f449138126fc26b5f3a5c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 166bbea04fd64826a5aae830ef8bfbae in term 1.
I20260926 01:10:02.769932 2274 raft_consensus.cc:2454] T ccc241eca243493a957eee8609bf8ad5 P 61c33320fe9844018afb635f9dee3641 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 166bbea04fd64826a5aae830ef8bfbae in term 1.
I20260926 01:10:02.770053 2347 leader_election.cc:302] T ccc241eca243493a957eee8609bf8ad5 P 166bbea04fd64826a5aae830ef8bfbae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 166bbea04fd64826a5aae830ef8bfbae, d73081e8b2f449138126fc26b5f3a5c2; no voters:
I20260926 01:10:02.770154 2626 raft_consensus.cc:2781] T ccc241eca243493a957eee8609bf8ad5 P 166bbea04fd64826a5aae830ef8bfbae [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:10:02.770327 2626 raft_consensus.cc:686] T ccc241eca243493a957eee8609bf8ad5 P 166bbea04fd64826a5aae830ef8bfbae [term 1 LEADER]: Becoming Leader. State: Replica: 166bbea04fd64826a5aae830ef8bfbae, State: Running, Role: LEADER
I20260926 01:10:02.770445 2626 consensus_queue.cc:237] T ccc241eca243493a957eee8609bf8ad5 P 166bbea04fd64826a5aae830ef8bfbae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "166bbea04fd64826a5aae830ef8bfbae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43727 } } peers { permanent_uuid: "d73081e8b2f449138126fc26b5f3a5c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36989 } } peers { permanent_uuid: "61c33320fe9844018afb635f9dee3641" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35639 } }
I20260926 01:10:02.771245 2117 catalog_manager.cc:5495] T ccc241eca243493a957eee8609bf8ad5 P 166bbea04fd64826a5aae830ef8bfbae reported cstate change: term changed from 0 to 1, leader changed from <none> to 166bbea04fd64826a5aae830ef8bfbae (127.0.98.2). New cstate: current_term: 1 leader_uuid: "166bbea04fd64826a5aae830ef8bfbae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "166bbea04fd64826a5aae830ef8bfbae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43727 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d73081e8b2f449138126fc26b5f3a5c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36989 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "61c33320fe9844018afb635f9dee3641" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35639 } health_report { overall_health: UNKNOWN } } }
Sep 26 01:10:02 dist-test-slave-7b22 krb5kdc[2072](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385002, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:10:02.809173 2166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:02.802701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49718 (local address 127.0.98.62:43913)
0926 01:10:02.802795 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:02.802797 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:02.802870 (+ 73us) server_negotiation.cc:407] Connection header received
0926 01:10:02.802916 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:02.802917 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:02.802926 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:02.802981 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:02.803418 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:02.803972 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:02.804692 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:02.804892 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:02.804954 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:02.807039 (+ 2085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:10:02.807040 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:10:02.807041 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:10:02.807051 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:10:02.808217 (+ 1166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:02.808555 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:02.808556 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:02.808557 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:02.808604 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:02.808737 (+ 133us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:02.808738 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:02.808738 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:02.808828 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:10:02.808871 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:02.809037 (+ 166us) server_negotiation.cc:299] Negotiation successful
0926 01:10:02.809101 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 01:10:02.809265 2637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:02.802625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43913 (local address 127.0.0.1:49718)
0926 01:10:02.802792 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:02.802802 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:02.802871 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:02.802993 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:02.802993 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:02.803277 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:10:02.803325 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:02.803327 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:02.804047 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:02.804047 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:02.804579 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:02.804580 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:02.804956 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:02.804957 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:02.805001 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:02.805499 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:10:02.805503 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:10:02.806926 (+ 1423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:10:02.808307 (+ 1381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:10:02.808315 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:10:02.808317 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:10:02.808505 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:10:02.808656 (+ 151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:10:02.808657 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:10:02.808657 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:10:02.808705 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:10:02.808901 (+ 196us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:10:02.808907 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:10:02.808954 (+ 47us) client_negotiation.cc:769] Sending connection context
0926 01:10:02.809018 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 01:10:02.809158 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"mutex_wait_us":46,"thread_start_us":87,"threads_started":1}
I20260926 01:10:02.811326 2117 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49718:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 01:10:02.811519 2117 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:10:02.812958 2254 tablet_service.cc:1467] Processing CreateTablet for tablet e3cb182d8f004ad7be9c1e56895d724a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3c7b3ae63f9649598f1a632368f7db79]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 01:10:02.812966 2394 tablet_service.cc:1467] Processing CreateTablet for tablet e3cb182d8f004ad7be9c1e56895d724a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3c7b3ae63f9649598f1a632368f7db79]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 01:10:02.812990 2520 tablet_service.cc:1467] Processing CreateTablet for tablet e3cb182d8f004ad7be9c1e56895d724a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3c7b3ae63f9649598f1a632368f7db79]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 01:10:02.813129 2520 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3cb182d8f004ad7be9c1e56895d724a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:10:02.813140 2394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3cb182d8f004ad7be9c1e56895d724a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:10:02.813140 2254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3cb182d8f004ad7be9c1e56895d724a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:10:02.814316 2621 tablet_bootstrap.cc:492] T e3cb182d8f004ad7be9c1e56895d724a P d73081e8b2f449138126fc26b5f3a5c2: Bootstrap starting.
I20260926 01:10:02.814723 2622 tablet_bootstrap.cc:492] T e3cb182d8f004ad7be9c1e56895d724a P 61c33320fe9844018afb635f9dee3641: Bootstrap starting.
I20260926 01:10:02.814841 2621 tablet_bootstrap.cc:654] T e3cb182d8f004ad7be9c1e56895d724a P d73081e8b2f449138126fc26b5f3a5c2: Neither blocks nor log segments found. Creating new log.
I20260926 01:10:02.815392 2622 tablet_bootstrap.cc:654] T e3cb182d8f004ad7be9c1e56895d724a P 61c33320fe9844018afb635f9dee3641: Neither blocks nor log segments found. Creating new log.
I20260926 01:10:02.815562 2621 tablet_bootstrap.cc:492] T e3cb182d8f004ad7be9c1e56895d724a P d73081e8b2f449138126fc26b5f3a5c2: No bootstrap required, opened a new log
I20260926 01:10:02.815620 2621 ts_tablet_manager.cc:1397] T e3cb182d8f004ad7be9c1e56895d724a P d73081e8b2f449138126fc26b5f3a5c2: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 01:10:02.815708 2620 tablet_bootstrap.cc:492] T e3cb182d8f004ad7be9c1e56895d724a P 166bbea04fd64826a5aae830ef8bfbae: Bootstrap starting.
I20260926 01:10:02.815783 2621 raft_consensus.cc:348] T e3cb182d8f004ad7be9c1e56895d724a P d73081e8b2f449138126fc26b5f3a5c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61c33320fe9844018afb635f9dee3641" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35639 } } peers { permanent_uuid: "166bbea04fd64826a5aae830ef8bfbae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43727 } } peers { permanent_uuid: "d73081e8b2f449138126fc26b5f3a5c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36989 } }
I20260926 01:10:02.815843 2621 raft_consensus.cc:374] T e3cb182d8f004ad7be9c1e56895d724a P d73081e8b2f449138126fc26b5f3a5c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:10:02.815856 2621 raft_consensus.cc:729] T e3cb182d8f004ad7be9c1e56895d724a P d73081e8b2f449138126fc26b5f3a5c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d73081e8b2f449138126fc26b5f3a5c2, State: Initialized, Role: FOLLOWER
I20260926 01:10:02.815891 2621 consensus_queue.cc:260] T e3cb182d8f004ad7be9c1e56895d724a P d73081e8b2f449138126fc26b5f3a5c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61c33320fe9844018afb635f9dee3641" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35639 } } peers { permanent_uuid: "166bbea04fd64826a5aae830ef8bfbae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43727 } } peers { permanent_uuid: "d73081e8b2f449138126fc26b5f3a5c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36989 } }
I20260926 01:10:02.815976 2621 ts_tablet_manager.cc:1428] T e3cb182d8f004ad7be9c1e56895d724a P d73081e8b2f449138126fc26b5f3a5c2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:02.816058 2622 tablet_bootstrap.cc:492] T e3cb182d8f004ad7be9c1e56895d724a P 61c33320fe9844018afb635f9dee3641: No bootstrap required, opened a new log
I20260926 01:10:02.816082 2620 tablet_bootstrap.cc:654] T e3cb182d8f004ad7be9c1e56895d724a P 166bbea04fd64826a5aae830ef8bfbae: Neither blocks nor log segments found. Creating new log.
I20260926 01:10:02.816098 2622 ts_tablet_manager.cc:1397] T e3cb182d8f004ad7be9c1e56895d724a P 61c33320fe9844018afb635f9dee3641: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260926 01:10:02.816156 2627 tablet_replica.cc:439] T e3cb182d8f004ad7be9c1e56895d724a P d73081e8b2f449138126fc26b5f3a5c2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61c33320fe9844018afb635f9dee3641" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35639 } } peers { permanent_uuid: "166bbea04fd64826a5aae830ef8bfbae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43727 } } peers { permanent_uuid: "d73081e8b2f449138126fc26b5f3a5c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36989 } } }
I20260926 01:10:02.816236 2627 tablet_replica.cc:442] T e3cb182d8f004ad7be9c1e56895d724a P d73081e8b2f449138126fc26b5f3a5c2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 01:10:02.816232 2622 raft_consensus.cc:348] T e3cb182d8f004ad7be9c1e56895d724a P 61c33320fe9844018afb635f9dee3641 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61c33320fe9844018afb635f9dee3641" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35639 } } peers { permanent_uuid: "166bbea04fd64826a5aae830ef8bfbae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43727 } } peers { permanent_uuid: "d73081e8b2f449138126fc26b5f3a5c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36989 } }
I20260926 01:10:02.816291 2622 raft_consensus.cc:374] T e3cb182d8f004ad7be9c1e56895d724a P 61c33320fe9844018afb635f9dee3641 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:10:02.816311 2622 raft_consensus.cc:729] T e3cb182d8f004ad7be9c1e56895d724a P 61c33320fe9844018afb635f9dee3641 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61c33320fe9844018afb635f9dee3641, State: Initialized, Role: FOLLOWER
I20260926 01:10:02.816351 2622 consensus_queue.cc:260] T e3cb182d8f004ad7be9c1e56895d724a P 61c33320fe9844018afb635f9dee3641 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61c33320fe9844018afb635f9dee3641" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35639 } } peers { permanent_uuid: "166bbea04fd64826a5aae830ef8bfbae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43727 } } peers { permanent_uuid: "d73081e8b2f449138126fc26b5f3a5c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36989 } }
I20260926 01:10:02.816442 2622 ts_tablet_manager.cc:1428] T e3cb182d8f004ad7be9c1e56895d724a P 61c33320fe9844018afb635f9dee3641: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:02.816658 2620 tablet_bootstrap.cc:492] T e3cb182d8f004ad7be9c1e56895d724a P 166bbea04fd64826a5aae830ef8bfbae: No bootstrap required, opened a new log
I20260926 01:10:02.816705 2620 ts_tablet_manager.cc:1397] T e3cb182d8f004ad7be9c1e56895d724a P 166bbea04fd64826a5aae830ef8bfbae: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 01:10:02.816733 2628 tablet_replica.cc:439] T e3cb182d8f004ad7be9c1e56895d724a P 61c33320fe9844018afb635f9dee3641: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61c33320fe9844018afb635f9dee3641" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35639 } } peers { permanent_uuid: "166bbea04fd64826a5aae830ef8bfbae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43727 } } peers { permanent_uuid: "d73081e8b2f449138126fc26b5f3a5c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36989 } } }
I20260926 01:10:02.816823 2628 tablet_replica.cc:442] T e3cb182d8f004ad7be9c1e56895d724a P 61c33320fe9844018afb635f9dee3641: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 01:10:02.816849 2620 raft_consensus.cc:348] T e3cb182d8f004ad7be9c1e56895d724a P 166bbea04fd64826a5aae830ef8bfbae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61c33320fe9844018afb635f9dee3641" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35639 } } peers { permanent_uuid: "166bbea04fd64826a5aae830ef8bfbae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43727 } } peers { permanent_uuid: "d73081e8b2f449138126fc26b5f3a5c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36989 } }
I20260926 01:10:02.816933 2620 raft_consensus.cc:374] T e3cb182d8f004ad7be9c1e56895d724a P 166bbea04fd64826a5aae830ef8bfbae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:10:02.816954 2620 raft_consensus.cc:729] T e3cb182d8f004ad7be9c1e56895d724a P 166bbea04fd64826a5aae830ef8bfbae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 166bbea04fd64826a5aae830ef8bfbae, State: Initialized, Role: FOLLOWER
I20260926 01:10:02.816995 2620 consensus_queue.cc:260] T e3cb182d8f004ad7be9c1e56895d724a P 166bbea04fd64826a5aae830ef8bfbae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61c33320fe9844018afb635f9dee3641" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35639 } } peers { permanent_uuid: "166bbea04fd64826a5aae830ef8bfbae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43727 } } peers { permanent_uuid: "d73081e8b2f449138126fc26b5f3a5c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36989 } }
I20260926 01:10:02.817087 2620 ts_tablet_manager.cc:1428] T e3cb182d8f004ad7be9c1e56895d724a P 166bbea04fd64826a5aae830ef8bfbae: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:02.817121 2626 tablet_replica.cc:439] T e3cb182d8f004ad7be9c1e56895d724a P 166bbea04fd64826a5aae830ef8bfbae: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61c33320fe9844018afb635f9dee3641" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35639 } } peers { permanent_uuid: "166bbea04fd64826a5aae830ef8bfbae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43727 } } peers { permanent_uuid: "d73081e8b2f449138126fc26b5f3a5c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36989 } } }
I20260926 01:10:02.817189 2626 tablet_replica.cc:442] T e3cb182d8f004ad7be9c1e56895d724a P 166bbea04fd64826a5aae830ef8bfbae: This TxnStatusTablet replica's current role is: FOLLOWER
W20260926 01:10:02.828702 2321 tablet.cc:2369] T ccc241eca243493a957eee8609bf8ad5 P 61c33320fe9844018afb635f9dee3641: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 01:10:02.830363 2627 raft_consensus.cc:482] T e3cb182d8f004ad7be9c1e56895d724a P d73081e8b2f449138126fc26b5f3a5c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:10:02.830475 2627 raft_consensus.cc:504] T e3cb182d8f004ad7be9c1e56895d724a P d73081e8b2f449138126fc26b5f3a5c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61c33320fe9844018afb635f9dee3641" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35639 } } peers { permanent_uuid: "166bbea04fd64826a5aae830ef8bfbae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43727 } } peers { permanent_uuid: "d73081e8b2f449138126fc26b5f3a5c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36989 } }
I20260926 01:10:02.830873 2627 leader_election.cc:288] T e3cb182d8f004ad7be9c1e56895d724a P d73081e8b2f449138126fc26b5f3a5c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 61c33320fe9844018afb635f9dee3641 (127.0.98.1:35639), 166bbea04fd64826a5aae830ef8bfbae (127.0.98.2:43727)
I20260926 01:10:02.833856 2619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:02.831100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36713 (local address 127.0.98.1:35639)
0926 01:10:02.831154 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:02.831156 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:02.831165 (+ 9us) server_negotiation.cc:407] Connection header received
0926 01:10:02.831232 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:02.831233 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:02.831242 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:02.831287 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:10:02.831498 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:02.832038 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:02.833150 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:02.833510 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:02.833569 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:02.833580 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:02.833715 (+ 135us) server_negotiation.cc:299] Negotiation successful
0926 01:10:02.833763 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 01:10:02.833856 2618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:02.831216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59481 (local address 127.0.98.2:43727)
0926 01:10:02.831277 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:02.831279 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:02.831381 (+ 102us) server_negotiation.cc:407] Connection header received
0926 01:10:02.831447 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:02.831448 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:02.831458 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:02.831499 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:10:02.831697 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:02.832162 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:02.833142 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:02.833510 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:02.833569 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:02.833581 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:02.833717 (+ 136us) server_negotiation.cc:299] Negotiation successful
0926 01:10:02.833761 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 01:10:02.833869 2493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:02.831047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35639 (local address 127.0.98.3:36713)
0926 01:10:02.831105 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:02.831118 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:02.831171 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:02.831337 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:02.831338 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:02.831339 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:10:02.831428 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:02.831430 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:02.832119 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:02.832120 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:02.833074 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:02.833077 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:02.833574 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:02.833575 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:02.833658 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:02.833661 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 01:10:02.833704 (+ 43us) client_negotiation.cc:240] Negotiation successful
0926 01:10:02.833754 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":46}
I20260926 01:10:02.833886 2641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:02.831155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43727 (local address 127.0.98.3:59481)
0926 01:10:02.831323 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:02.831332 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:02.831376 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:02.831514 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:02.831514 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:02.831516 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:10:02.831633 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:02.831635 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:02.832232 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:02.832233 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:02.833053 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:02.833056 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:02.833598 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:02.833599 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:02.833658 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:02.833661 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 01:10:02.833705 (+ 44us) client_negotiation.cc:240] Negotiation successful
0926 01:10:02.833753 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"mutex_wait_us":25,"thread_start_us":104,"threads_started":1}
I20260926 01:10:02.834169 2274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3cb182d8f004ad7be9c1e56895d724a" candidate_uuid: "d73081e8b2f449138126fc26b5f3a5c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61c33320fe9844018afb635f9dee3641" is_pre_election: true
I20260926 01:10:02.834201 2414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3cb182d8f004ad7be9c1e56895d724a" candidate_uuid: "d73081e8b2f449138126fc26b5f3a5c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "166bbea04fd64826a5aae830ef8bfbae" is_pre_election: true
I20260926 01:10:02.834261 2274 raft_consensus.cc:2454] T e3cb182d8f004ad7be9c1e56895d724a P 61c33320fe9844018afb635f9dee3641 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d73081e8b2f449138126fc26b5f3a5c2 in term 0.
I20260926 01:10:02.834306 2414 raft_consensus.cc:2454] T e3cb182d8f004ad7be9c1e56895d724a P 166bbea04fd64826a5aae830ef8bfbae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d73081e8b2f449138126fc26b5f3a5c2 in term 0.
I20260926 01:10:02.834415 2488 leader_election.cc:302] T e3cb182d8f004ad7be9c1e56895d724a P d73081e8b2f449138126fc26b5f3a5c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61c33320fe9844018afb635f9dee3641, d73081e8b2f449138126fc26b5f3a5c2; no voters:
I20260926 01:10:02.834558 2627 raft_consensus.cc:2781] T e3cb182d8f004ad7be9c1e56895d724a P d73081e8b2f449138126fc26b5f3a5c2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:10:02.834610 2627 raft_consensus.cc:482] T e3cb182d8f004ad7be9c1e56895d724a P d73081e8b2f449138126fc26b5f3a5c2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:10:02.834632 2627 raft_consensus.cc:3037] T e3cb182d8f004ad7be9c1e56895d724a P d73081e8b2f449138126fc26b5f3a5c2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:10:02.835331 2627 raft_consensus.cc:504] T e3cb182d8f004ad7be9c1e56895d724a P d73081e8b2f449138126fc26b5f3a5c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61c33320fe9844018afb635f9dee3641" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35639 } } peers { permanent_uuid: "166bbea04fd64826a5aae830ef8bfbae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43727 } } peers { permanent_uuid: "d73081e8b2f449138126fc26b5f3a5c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36989 } }
I20260926 01:10:02.835448 2627 leader_election.cc:288] T e3cb182d8f004ad7be9c1e56895d724a P d73081e8b2f449138126fc26b5f3a5c2 [CANDIDATE]: Term 1 election: Requested vote from peers 61c33320fe9844018afb635f9dee3641 (127.0.98.1:35639), 166bbea04fd64826a5aae830ef8bfbae (127.0.98.2:43727)
I20260926 01:10:02.835618 2274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3cb182d8f004ad7be9c1e56895d724a" candidate_uuid: "d73081e8b2f449138126fc26b5f3a5c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61c33320fe9844018afb635f9dee3641"
I20260926 01:10:02.835698 2274 raft_consensus.cc:3037] T e3cb182d8f004ad7be9c1e56895d724a P 61c33320fe9844018afb635f9dee3641 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:10:02.835762 2414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3cb182d8f004ad7be9c1e56895d724a" candidate_uuid: "d73081e8b2f449138126fc26b5f3a5c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "166bbea04fd64826a5aae830ef8bfbae"
I20260926 01:10:02.835834 2414 raft_consensus.cc:3037] T e3cb182d8f004ad7be9c1e56895d724a P 166bbea04fd64826a5aae830ef8bfbae [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:10:02.836378 2274 raft_consensus.cc:2454] T e3cb182d8f004ad7be9c1e56895d724a P 61c33320fe9844018afb635f9dee3641 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d73081e8b2f449138126fc26b5f3a5c2 in term 1.
I20260926 01:10:02.836519 2414 raft_consensus.cc:2454] T e3cb182d8f004ad7be9c1e56895d724a P 166bbea04fd64826a5aae830ef8bfbae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d73081e8b2f449138126fc26b5f3a5c2 in term 1.
I20260926 01:10:02.836541 2488 leader_election.cc:302] T e3cb182d8f004ad7be9c1e56895d724a P d73081e8b2f449138126fc26b5f3a5c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61c33320fe9844018afb635f9dee3641, d73081e8b2f449138126fc26b5f3a5c2; no voters:
I20260926 01:10:02.836630 2627 raft_consensus.cc:2781] T e3cb182d8f004ad7be9c1e56895d724a P d73081e8b2f449138126fc26b5f3a5c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:10:02.836786 2627 raft_consensus.cc:686] T e3cb182d8f004ad7be9c1e56895d724a P d73081e8b2f449138126fc26b5f3a5c2 [term 1 LEADER]: Becoming Leader. State: Replica: d73081e8b2f449138126fc26b5f3a5c2, State: Running, Role: LEADER
I20260926 01:10:02.836884 2627 consensus_queue.cc:237] T e3cb182d8f004ad7be9c1e56895d724a P d73081e8b2f449138126fc26b5f3a5c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61c33320fe9844018afb635f9dee3641" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35639 } } peers { permanent_uuid: "166bbea04fd64826a5aae830ef8bfbae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43727 } } peers { permanent_uuid: "d73081e8b2f449138126fc26b5f3a5c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36989 } }
I20260926 01:10:02.837348 2642 tablet_replica.cc:439] T e3cb182d8f004ad7be9c1e56895d724a P d73081e8b2f449138126fc26b5f3a5c2: TxnStatusTablet state changed. Reason: New leader d73081e8b2f449138126fc26b5f3a5c2. Latest consensus state: current_term: 1 leader_uuid: "d73081e8b2f449138126fc26b5f3a5c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61c33320fe9844018afb635f9dee3641" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35639 } } peers { permanent_uuid: "166bbea04fd64826a5aae830ef8bfbae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43727 } } peers { permanent_uuid: "d73081e8b2f449138126fc26b5f3a5c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36989 } } }
I20260926 01:10:02.837471 2642 tablet_replica.cc:442] T e3cb182d8f004ad7be9c1e56895d724a P d73081e8b2f449138126fc26b5f3a5c2: This TxnStatusTablet replica's current role is: LEADER
I20260926 01:10:02.837643 2644 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 01:10:02.837752 2644 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 01:10:02.837726 2116 catalog_manager.cc:5495] T e3cb182d8f004ad7be9c1e56895d724a P d73081e8b2f449138126fc26b5f3a5c2 reported cstate change: term changed from 0 to 1, leader changed from <none> to d73081e8b2f449138126fc26b5f3a5c2 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "d73081e8b2f449138126fc26b5f3a5c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61c33320fe9844018afb635f9dee3641" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "166bbea04fd64826a5aae830ef8bfbae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43727 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d73081e8b2f449138126fc26b5f3a5c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36989 } health_report { overall_health: HEALTHY } } }
I20260926 01:10:02.847442 2616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:02.844659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42108 (local address 127.0.98.3:36989)
0926 01:10:02.844720 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:02.844722 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:02.844733 (+ 11us) server_negotiation.cc:407] Connection header received
0926 01:10:02.844764 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:02.844765 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:02.844785 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:02.844857 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:10:02.844948 (+ 91us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:02.845730 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:02.846625 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:02.846879 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:02.846938 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:02.847120 (+ 182us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:10:02.847227 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:10:02.847295 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:02.847338 (+ 43us) server_negotiation.cc:299] Negotiation successful
0926 01:10:02.847362 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 01:10:02.847456 2637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:02.844539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36989 (local address 127.0.0.1:42108)
0926 01:10:02.844623 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:02.844636 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:02.844688 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:02.844857 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:02.844858 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:02.844859 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 01:10:02.844905 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:02.844908 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:02.845807 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:02.845807 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:02.846477 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:02.846480 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:02.846939 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:02.846940 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:02.847013 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:02.847030 (+ 17us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 01:10:02.847279 (+ 249us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 01:10:02.847287 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 01:10:02.847333 (+ 46us) client_negotiation.cc:240] Negotiation successful
0926 01:10:02.847392 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260926 01:10:02.849186 2414 raft_consensus.cc:1261] T e3cb182d8f004ad7be9c1e56895d724a P 166bbea04fd64826a5aae830ef8bfbae [term 1 FOLLOWER]: Refusing update from remote peer d73081e8b2f449138126fc26b5f3a5c2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:10:02.849222 2274 raft_consensus.cc:1261] T e3cb182d8f004ad7be9c1e56895d724a P 61c33320fe9844018afb635f9dee3641 [term 1 FOLLOWER]: Refusing update from remote peer d73081e8b2f449138126fc26b5f3a5c2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:10:02.849529 2642 consensus_queue.cc:1035] T e3cb182d8f004ad7be9c1e56895d724a P d73081e8b2f449138126fc26b5f3a5c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "166bbea04fd64826a5aae830ef8bfbae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:10:02.849663 2627 consensus_queue.cc:1035] T e3cb182d8f004ad7be9c1e56895d724a P d73081e8b2f449138126fc26b5f3a5c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61c33320fe9844018afb635f9dee3641" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:10:02.850849 2626 tablet_replica.cc:439] T e3cb182d8f004ad7be9c1e56895d724a P 166bbea04fd64826a5aae830ef8bfbae: TxnStatusTablet state changed. Reason: New leader d73081e8b2f449138126fc26b5f3a5c2. Latest consensus state: current_term: 1 leader_uuid: "d73081e8b2f449138126fc26b5f3a5c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61c33320fe9844018afb635f9dee3641" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35639 } } peers { permanent_uuid: "166bbea04fd64826a5aae830ef8bfbae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43727 } } peers { permanent_uuid: "d73081e8b2f449138126fc26b5f3a5c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36989 } } }
I20260926 01:10:02.850871 2628 tablet_replica.cc:439] T e3cb182d8f004ad7be9c1e56895d724a P 61c33320fe9844018afb635f9dee3641: TxnStatusTablet state changed. Reason: New leader d73081e8b2f449138126fc26b5f3a5c2. Latest consensus state: current_term: 1 leader_uuid: "d73081e8b2f449138126fc26b5f3a5c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61c33320fe9844018afb635f9dee3641" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35639 } } peers { permanent_uuid: "166bbea04fd64826a5aae830ef8bfbae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43727 } } peers { permanent_uuid: "d73081e8b2f449138126fc26b5f3a5c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36989 } } }
I20260926 01:10:02.850946 2626 tablet_replica.cc:442] T e3cb182d8f004ad7be9c1e56895d724a P 166bbea04fd64826a5aae830ef8bfbae: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 01:10:02.850956 2628 tablet_replica.cc:442] T e3cb182d8f004ad7be9c1e56895d724a P 61c33320fe9844018afb635f9dee3641: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 01:10:02.851893 2642 tablet_replica.cc:439] T e3cb182d8f004ad7be9c1e56895d724a P d73081e8b2f449138126fc26b5f3a5c2: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d73081e8b2f449138126fc26b5f3a5c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61c33320fe9844018afb635f9dee3641" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35639 } } peers { permanent_uuid: "166bbea04fd64826a5aae830ef8bfbae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43727 } } peers { permanent_uuid: "d73081e8b2f449138126fc26b5f3a5c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36989 } } }
I20260926 01:10:02.851976 2642 tablet_replica.cc:442] T e3cb182d8f004ad7be9c1e56895d724a P d73081e8b2f449138126fc26b5f3a5c2: This TxnStatusTablet replica's current role is: LEADER
I20260926 01:10:02.852041 2628 tablet_replica.cc:439] T e3cb182d8f004ad7be9c1e56895d724a P 61c33320fe9844018afb635f9dee3641: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d73081e8b2f449138126fc26b5f3a5c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61c33320fe9844018afb635f9dee3641" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35639 } } peers { permanent_uuid: "166bbea04fd64826a5aae830ef8bfbae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43727 } } peers { permanent_uuid: "d73081e8b2f449138126fc26b5f3a5c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36989 } } }
I20260926 01:10:02.852116 2628 tablet_replica.cc:442] T e3cb182d8f004ad7be9c1e56895d724a P 61c33320fe9844018afb635f9dee3641: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 01:10:02.852257 2647 mvcc.cc:205] Tried to move back new op lower bound from 7333416971667800064 to 7333416971620548608. Current Snapshot: MvccSnapshot[applied={T|T < 7333416971667800064}]
I20260926 01:10:02.852646 2642 tablet_replica.cc:439] T e3cb182d8f004ad7be9c1e56895d724a P d73081e8b2f449138126fc26b5f3a5c2: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d73081e8b2f449138126fc26b5f3a5c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61c33320fe9844018afb635f9dee3641" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35639 } } peers { permanent_uuid: "166bbea04fd64826a5aae830ef8bfbae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43727 } } peers { permanent_uuid: "d73081e8b2f449138126fc26b5f3a5c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36989 } } }
I20260926 01:10:02.852723 2642 tablet_replica.cc:442] T e3cb182d8f004ad7be9c1e56895d724a P d73081e8b2f449138126fc26b5f3a5c2: This TxnStatusTablet replica's current role is: LEADER
I20260926 01:10:02.853088 2626 tablet_replica.cc:439] T e3cb182d8f004ad7be9c1e56895d724a P 166bbea04fd64826a5aae830ef8bfbae: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d73081e8b2f449138126fc26b5f3a5c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61c33320fe9844018afb635f9dee3641" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35639 } } peers { permanent_uuid: "166bbea04fd64826a5aae830ef8bfbae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43727 } } peers { permanent_uuid: "d73081e8b2f449138126fc26b5f3a5c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36989 } } }
I20260926 01:10:02.853170 2626 tablet_replica.cc:442] T e3cb182d8f004ad7be9c1e56895d724a P 166bbea04fd64826a5aae830ef8bfbae: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 01:10:02.853654 2649 mvcc.cc:205] Tried to move back new op lower bound from 7333416971667800064 to 7333416971620548608. Current Snapshot: MvccSnapshot[applied={T|T < 7333416971667800064}]
I20260926 01:10:02.853952 2648 mvcc.cc:205] Tried to move back new op lower bound from 7333416971667800064 to 7333416971620548608. Current Snapshot: MvccSnapshot[applied={T|T < 7333416971667800064}]
I20260926 01:10:02.856786 2618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:02.853890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38976 (local address 127.0.98.2:43727)
0926 01:10:02.853947 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:02.853949 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:02.854051 (+ 102us) server_negotiation.cc:407] Connection header received
0926 01:10:02.854082 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:02.854083 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:02.854100 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:02.854159 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:10:02.854324 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:02.855059 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:02.855867 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:02.856162 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:02.856241 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:02.856413 (+ 172us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:10:02.856539 (+ 126us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:10:02.856617 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:02.856687 (+ 70us) server_negotiation.cc:299] Negotiation successful
0926 01:10:02.856710 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 01:10:02.862044 2616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:02.859444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44105 (local address 127.0.98.3:36989)
0926 01:10:02.859485 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:02.859486 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:02.859495 (+ 9us) server_negotiation.cc:407] Connection header received
0926 01:10:02.859564 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:02.859565 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:02.859585 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:02.859629 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:10:02.859751 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:02.860216 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:02.861327 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:02.861520 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:02.861594 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:02.861791 (+ 197us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:10:02.861849 (+ 58us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:10:02.861902 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:02.861961 (+ 59us) server_negotiation.cc:299] Negotiation successful
0926 01:10:02.861980 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 01:10:02.862092 2466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:02.859383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36989 (local address 127.0.98.2:44105)
0926 01:10:02.859449 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:02.859461 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:02.859506 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:02.859632 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:02.859633 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:02.859634 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 01:10:02.859694 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:02.859696 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:02.860278 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:02.860279 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:02.861202 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:02.861205 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:02.861597 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:02.861598 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:02.861661 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:02.861682 (+ 21us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 01:10:02.861902 (+ 220us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 01:10:02.861904 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 01:10:02.861955 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 01:10:02.862005 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260926 01:10:02.864516 2274 raft_consensus.cc:1261] T ccc241eca243493a957eee8609bf8ad5 P 61c33320fe9844018afb635f9dee3641 [term 1 FOLLOWER]: Refusing update from remote peer 166bbea04fd64826a5aae830ef8bfbae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:10:02.864614 2554 raft_consensus.cc:1261] T ccc241eca243493a957eee8609bf8ad5 P d73081e8b2f449138126fc26b5f3a5c2 [term 1 FOLLOWER]: Refusing update from remote peer 166bbea04fd64826a5aae830ef8bfbae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:10:02.864923 2630 consensus_queue.cc:1035] T ccc241eca243493a957eee8609bf8ad5 P 166bbea04fd64826a5aae830ef8bfbae [LEADER]: Connected to new peer: Peer: permanent_uuid: "d73081e8b2f449138126fc26b5f3a5c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:10:02.865057 2626 consensus_queue.cc:1035] T ccc241eca243493a957eee8609bf8ad5 P 166bbea04fd64826a5aae830ef8bfbae [LEADER]: Connected to new peer: Peer: permanent_uuid: "61c33320fe9844018afb635f9dee3641" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:10:02.874392 2606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:02.871528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43727 (local address 127.0.98.3:53419)
0926 01:10:02.871599 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:02.871610 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:02.871648 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:02.871764 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:02.871764 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:02.871765 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 01:10:02.871823 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:02.871824 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:02.872726 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:02.872727 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:02.873557 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:02.873561 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:02.874038 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:02.874039 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:02.874090 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:02.874106 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 01:10:02.874265 (+ 159us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 01:10:02.874266 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 01:10:02.874295 (+ 29us) client_negotiation.cc:240] Negotiation successful
0926 01:10:02.874329 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260926 01:10:02.874406 2618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:02.871599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53419 (local address 127.0.98.2:43727)
0926 01:10:02.871637 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:02.871639 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:02.871648 (+ 9us) server_negotiation.cc:407] Connection header received
0926 01:10:02.871691 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:02.871691 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:02.871702 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:02.871761 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:10:02.871861 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:02.872605 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:02.873635 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:02.873967 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:02.874030 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:02.874171 (+ 141us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:10:02.874207 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:10:02.874265 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:02.874300 (+ 35us) server_negotiation.cc:299] Negotiation successful
0926 01:10:02.874323 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 01:10:02.883968 2115 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49706:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:10:02.884105 2115 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49706:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:10:02.884950 2115 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 43da41c6d9f34b998e1c66111660121f: Sending DeleteTablet for 3 replicas of tablet ccc241eca243493a957eee8609bf8ad5
I20260926 01:10:02.885355 2254 tablet_service.cc:1514] Processing DeleteTablet for tablet ccc241eca243493a957eee8609bf8ad5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:10:02 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53100
I20260926 01:10:02.885617 2115 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49706:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 01:10:02.885713 2394 tablet_service.cc:1514] Processing DeleteTablet for tablet ccc241eca243493a957eee8609bf8ad5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:10:02 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38970
W20260926 01:10:02.885784 2115 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:10:02.886317 2664 tablet_replica.cc:330] T ccc241eca243493a957eee8609bf8ad5 P 61c33320fe9844018afb635f9dee3641: stopping tablet replica
I20260926 01:10:02.886539 2664 raft_consensus.cc:2229] T ccc241eca243493a957eee8609bf8ad5 P 61c33320fe9844018afb635f9dee3641 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:10:02.886556 2520 tablet_service.cc:1514] Processing DeleteTablet for tablet ccc241eca243493a957eee8609bf8ad5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:10:02 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42100
I20260926 01:10:02.886695 2664 raft_consensus.cc:2258] T ccc241eca243493a957eee8609bf8ad5 P 61c33320fe9844018afb635f9dee3641 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:10:02.887333 2666 tablet_replica.cc:330] T ccc241eca243493a957eee8609bf8ad5 P d73081e8b2f449138126fc26b5f3a5c2: stopping tablet replica
I20260926 01:10:02.887511 2666 raft_consensus.cc:2229] T ccc241eca243493a957eee8609bf8ad5 P d73081e8b2f449138126fc26b5f3a5c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:10:02.887609 2666 raft_consensus.cc:2258] T ccc241eca243493a957eee8609bf8ad5 P d73081e8b2f449138126fc26b5f3a5c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:10:02.887826 2254 tablet_service.cc:1467] Processing CreateTablet for tablet 6fca2e9e79da4360893bf0cc0e1101fd (DEFAULT_TABLE table=test-table [id=1d2003007f8244c496a059cfcedfde53]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:10:02.887956 2254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fca2e9e79da4360893bf0cc0e1101fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:10:02.888005 2394 tablet_service.cc:1467] Processing CreateTablet for tablet 6fca2e9e79da4360893bf0cc0e1101fd (DEFAULT_TABLE table=test-table [id=1d2003007f8244c496a059cfcedfde53]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:10:02.888125 2394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fca2e9e79da4360893bf0cc0e1101fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:10:02.889506 2622 tablet_bootstrap.cc:492] T 6fca2e9e79da4360893bf0cc0e1101fd P 61c33320fe9844018afb635f9dee3641: Bootstrap starting.
I20260926 01:10:02.889545 2520 tablet_service.cc:1467] Processing CreateTablet for tablet 6fca2e9e79da4360893bf0cc0e1101fd (DEFAULT_TABLE table=test-table [id=1d2003007f8244c496a059cfcedfde53]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:10:02.889670 2520 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fca2e9e79da4360893bf0cc0e1101fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:10:02.889935 2622 tablet_bootstrap.cc:654] T 6fca2e9e79da4360893bf0cc0e1101fd P 61c33320fe9844018afb635f9dee3641: Neither blocks nor log segments found. Creating new log.
I20260926 01:10:02.891059 2620 tablet_bootstrap.cc:492] T 6fca2e9e79da4360893bf0cc0e1101fd P 166bbea04fd64826a5aae830ef8bfbae: Bootstrap starting.
I20260926 01:10:02.891212 2666 ts_tablet_manager.cc:1905] T ccc241eca243493a957eee8609bf8ad5 P d73081e8b2f449138126fc26b5f3a5c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:10:02.891491 2620 tablet_bootstrap.cc:654] T 6fca2e9e79da4360893bf0cc0e1101fd P 166bbea04fd64826a5aae830ef8bfbae: Neither blocks nor log segments found. Creating new log.
I20260926 01:10:02.891885 2621 tablet_bootstrap.cc:492] T 6fca2e9e79da4360893bf0cc0e1101fd P d73081e8b2f449138126fc26b5f3a5c2: Bootstrap starting.
I20260926 01:10:02.892280 2621 tablet_bootstrap.cc:654] T 6fca2e9e79da4360893bf0cc0e1101fd P d73081e8b2f449138126fc26b5f3a5c2: Neither blocks nor log segments found. Creating new log.
I20260926 01:10:02.893210 2664 ts_tablet_manager.cc:1905] T ccc241eca243493a957eee8609bf8ad5 P 61c33320fe9844018afb635f9dee3641: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:10:02.893471 2666 ts_tablet_manager.cc:1918] T ccc241eca243493a957eee8609bf8ad5 P d73081e8b2f449138126fc26b5f3a5c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 01:10:02.893594 2666 log.cc:1196] T ccc241eca243493a957eee8609bf8ad5 P d73081e8b2f449138126fc26b5f3a5c2: Deleting WAL directory at /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384993574762-392-0/minicluster-data/ts-2/wal/wals/ccc241eca243493a957eee8609bf8ad5
I20260926 01:10:02.893929 2666 ts_tablet_manager.cc:1939] T ccc241eca243493a957eee8609bf8ad5 P d73081e8b2f449138126fc26b5f3a5c2: Deleting consensus metadata
I20260926 01:10:02.894483 2103 catalog_manager.cc:4841] TS d73081e8b2f449138126fc26b5f3a5c2 (127.0.98.3:36989): tablet ccc241eca243493a957eee8609bf8ad5 (table test-table [id=beabdfb270e849c38f09bbbe1de4a838]) successfully deleted
I20260926 01:10:02.894932 2664 ts_tablet_manager.cc:1918] T ccc241eca243493a957eee8609bf8ad5 P 61c33320fe9844018afb635f9dee3641: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 01:10:02.895087 2664 log.cc:1196] T ccc241eca243493a957eee8609bf8ad5 P 61c33320fe9844018afb635f9dee3641: Deleting WAL directory at /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384993574762-392-0/minicluster-data/ts-0/wal/wals/ccc241eca243493a957eee8609bf8ad5
I20260926 01:10:02.895357 2622 tablet_bootstrap.cc:492] T 6fca2e9e79da4360893bf0cc0e1101fd P 61c33320fe9844018afb635f9dee3641: No bootstrap required, opened a new log
I20260926 01:10:02.895404 2622 ts_tablet_manager.cc:1397] T 6fca2e9e79da4360893bf0cc0e1101fd P 61c33320fe9844018afb635f9dee3641: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.001s
I20260926 01:10:02.895582 2622 raft_consensus.cc:348] T 6fca2e9e79da4360893bf0cc0e1101fd P 61c33320fe9844018afb635f9dee3641 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61c33320fe9844018afb635f9dee3641" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35639 } } peers { permanent_uuid: "166bbea04fd64826a5aae830ef8bfbae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43727 } } peers { permanent_uuid: "d73081e8b2f449138126fc26b5f3a5c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36989 } }
I20260926 01:10:02.895668 2622 raft_consensus.cc:374] T 6fca2e9e79da4360893bf0cc0e1101fd P 61c33320fe9844018afb635f9dee3641 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:10:02.895689 2622 raft_consensus.cc:729] T 6fca2e9e79da4360893bf0cc0e1101fd P 61c33320fe9844018afb635f9dee3641 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61c33320fe9844018afb635f9dee3641, State: Initialized, Role: FOLLOWER
I20260926 01:10:02.895737 2622 consensus_queue.cc:260] T 6fca2e9e79da4360893bf0cc0e1101fd P 61c33320fe9844018afb635f9dee3641 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61c33320fe9844018afb635f9dee3641" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35639 } } peers { permanent_uuid: "166bbea04fd64826a5aae830ef8bfbae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43727 } } peers { permanent_uuid: "d73081e8b2f449138126fc26b5f3a5c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36989 } }
I20260926 01:10:02.895833 2622 ts_tablet_manager.cc:1428] T 6fca2e9e79da4360893bf0cc0e1101fd P 61c33320fe9844018afb635f9dee3641: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:02.895392 2664 ts_tablet_manager.cc:1939] T ccc241eca243493a957eee8609bf8ad5 P 61c33320fe9844018afb635f9dee3641: Deleting consensus metadata
I20260926 01:10:02.896572 2104 catalog_manager.cc:4841] TS 61c33320fe9844018afb635f9dee3641 (127.0.98.1:35639): tablet ccc241eca243493a957eee8609bf8ad5 (table test-table [id=beabdfb270e849c38f09bbbe1de4a838]) successfully deleted
I20260926 01:10:02.896642 2665 tablet_replica.cc:330] T ccc241eca243493a957eee8609bf8ad5 P 166bbea04fd64826a5aae830ef8bfbae: stopping tablet replica
I20260926 01:10:02.896790 2620 tablet_bootstrap.cc:492] T 6fca2e9e79da4360893bf0cc0e1101fd P 166bbea04fd64826a5aae830ef8bfbae: No bootstrap required, opened a new log
I20260926 01:10:02.896837 2620 ts_tablet_manager.cc:1397] T 6fca2e9e79da4360893bf0cc0e1101fd P 166bbea04fd64826a5aae830ef8bfbae: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260926 01:10:02.896914 2665 raft_consensus.cc:2229] T ccc241eca243493a957eee8609bf8ad5 P 166bbea04fd64826a5aae830ef8bfbae [term 1 LEADER]: Raft consensus shutting down.
I20260926 01:10:02.897044 2665 raft_consensus.cc:2258] T ccc241eca243493a957eee8609bf8ad5 P 166bbea04fd64826a5aae830ef8bfbae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:10:02.896984 2620 raft_consensus.cc:348] T 6fca2e9e79da4360893bf0cc0e1101fd P 166bbea04fd64826a5aae830ef8bfbae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61c33320fe9844018afb635f9dee3641" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35639 } } peers { permanent_uuid: "166bbea04fd64826a5aae830ef8bfbae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43727 } } peers { permanent_uuid: "d73081e8b2f449138126fc26b5f3a5c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36989 } }
I20260926 01:10:02.897166 2620 raft_consensus.cc:374] T 6fca2e9e79da4360893bf0cc0e1101fd P 166bbea04fd64826a5aae830ef8bfbae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:10:02.897186 2620 raft_consensus.cc:729] T 6fca2e9e79da4360893bf0cc0e1101fd P 166bbea04fd64826a5aae830ef8bfbae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 166bbea04fd64826a5aae830ef8bfbae, State: Initialized, Role: FOLLOWER
I20260926 01:10:02.897234 2620 consensus_queue.cc:260] T 6fca2e9e79da4360893bf0cc0e1101fd P 166bbea04fd64826a5aae830ef8bfbae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61c33320fe9844018afb635f9dee3641" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35639 } } peers { permanent_uuid: "166bbea04fd64826a5aae830ef8bfbae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43727 } } peers { permanent_uuid: "d73081e8b2f449138126fc26b5f3a5c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36989 } }
I20260926 01:10:02.897330 2620 ts_tablet_manager.cc:1428] T 6fca2e9e79da4360893bf0cc0e1101fd P 166bbea04fd64826a5aae830ef8bfbae: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 01:10:02.897801 2621 tablet_bootstrap.cc:492] T 6fca2e9e79da4360893bf0cc0e1101fd P d73081e8b2f449138126fc26b5f3a5c2: No bootstrap required, opened a new log
I20260926 01:10:02.897873 2621 ts_tablet_manager.cc:1397] T 6fca2e9e79da4360893bf0cc0e1101fd P d73081e8b2f449138126fc26b5f3a5c2: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260926 01:10:02.897923 2665 ts_tablet_manager.cc:1905] T ccc241eca243493a957eee8609bf8ad5 P 166bbea04fd64826a5aae830ef8bfbae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:10:02.898017 2621 raft_consensus.cc:348] T 6fca2e9e79da4360893bf0cc0e1101fd P d73081e8b2f449138126fc26b5f3a5c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61c33320fe9844018afb635f9dee3641" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35639 } } peers { permanent_uuid: "166bbea04fd64826a5aae830ef8bfbae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43727 } } peers { permanent_uuid: "d73081e8b2f449138126fc26b5f3a5c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36989 } }
I20260926 01:10:02.898094 2621 raft_consensus.cc:374] T 6fca2e9e79da4360893bf0cc0e1101fd P d73081e8b2f449138126fc26b5f3a5c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:10:02.898114 2621 raft_consensus.cc:729] T 6fca2e9e79da4360893bf0cc0e1101fd P d73081e8b2f449138126fc26b5f3a5c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d73081e8b2f449138126fc26b5f3a5c2, State: Initialized, Role: FOLLOWER
I20260926 01:10:02.898172 2621 consensus_queue.cc:260] T 6fca2e9e79da4360893bf0cc0e1101fd P d73081e8b2f449138126fc26b5f3a5c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61c33320fe9844018afb635f9dee3641" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35639 } } peers { permanent_uuid: "166bbea04fd64826a5aae830ef8bfbae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43727 } } peers { permanent_uuid: "d73081e8b2f449138126fc26b5f3a5c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36989 } }
I20260926 01:10:02.898255 2621 ts_tablet_manager.cc:1428] T 6fca2e9e79da4360893bf0cc0e1101fd P d73081e8b2f449138126fc26b5f3a5c2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:02.899573 2665 ts_tablet_manager.cc:1918] T ccc241eca243493a957eee8609bf8ad5 P 166bbea04fd64826a5aae830ef8bfbae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
W20260926 01:10:02.899611 2601 tablet.cc:2369] T 6fca2e9e79da4360893bf0cc0e1101fd P d73081e8b2f449138126fc26b5f3a5c2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 01:10:02.899708 2665 log.cc:1196] T ccc241eca243493a957eee8609bf8ad5 P 166bbea04fd64826a5aae830ef8bfbae: Deleting WAL directory at /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384993574762-392-0/minicluster-data/ts-1/wal/wals/ccc241eca243493a957eee8609bf8ad5
I20260926 01:10:02.899950 2665 ts_tablet_manager.cc:1939] T ccc241eca243493a957eee8609bf8ad5 P 166bbea04fd64826a5aae830ef8bfbae: Deleting consensus metadata
I20260926 01:10:02.900377 2104 catalog_manager.cc:4841] TS 166bbea04fd64826a5aae830ef8bfbae (127.0.98.2:43727): tablet ccc241eca243493a957eee8609bf8ad5 (table test-table [id=beabdfb270e849c38f09bbbe1de4a838]) successfully deleted
I20260926 01:10:03.003896 2660 raft_consensus.cc:482] T 6fca2e9e79da4360893bf0cc0e1101fd P d73081e8b2f449138126fc26b5f3a5c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:10:03.004020 2660 raft_consensus.cc:504] T 6fca2e9e79da4360893bf0cc0e1101fd P d73081e8b2f449138126fc26b5f3a5c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61c33320fe9844018afb635f9dee3641" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35639 } } peers { permanent_uuid: "166bbea04fd64826a5aae830ef8bfbae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43727 } } peers { permanent_uuid: "d73081e8b2f449138126fc26b5f3a5c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36989 } }
I20260926 01:10:03.004182 2660 leader_election.cc:288] T 6fca2e9e79da4360893bf0cc0e1101fd P d73081e8b2f449138126fc26b5f3a5c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 61c33320fe9844018afb635f9dee3641 (127.0.98.1:35639), 166bbea04fd64826a5aae830ef8bfbae (127.0.98.2:43727)
I20260926 01:10:03.004429 2274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fca2e9e79da4360893bf0cc0e1101fd" candidate_uuid: "d73081e8b2f449138126fc26b5f3a5c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61c33320fe9844018afb635f9dee3641" is_pre_election: true
I20260926 01:10:03.004468 2414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fca2e9e79da4360893bf0cc0e1101fd" candidate_uuid: "d73081e8b2f449138126fc26b5f3a5c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "166bbea04fd64826a5aae830ef8bfbae" is_pre_election: true
I20260926 01:10:03.004544 2414 raft_consensus.cc:2454] T 6fca2e9e79da4360893bf0cc0e1101fd P 166bbea04fd64826a5aae830ef8bfbae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d73081e8b2f449138126fc26b5f3a5c2 in term 0.
I20260926 01:10:03.004544 2274 raft_consensus.cc:2454] T 6fca2e9e79da4360893bf0cc0e1101fd P 61c33320fe9844018afb635f9dee3641 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d73081e8b2f449138126fc26b5f3a5c2 in term 0.
I20260926 01:10:03.004753 2488 leader_election.cc:302] T 6fca2e9e79da4360893bf0cc0e1101fd P d73081e8b2f449138126fc26b5f3a5c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61c33320fe9844018afb635f9dee3641, d73081e8b2f449138126fc26b5f3a5c2; no voters:
I20260926 01:10:03.004858 2660 raft_consensus.cc:2781] T 6fca2e9e79da4360893bf0cc0e1101fd P d73081e8b2f449138126fc26b5f3a5c2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:10:03.004900 2660 raft_consensus.cc:482] T 6fca2e9e79da4360893bf0cc0e1101fd P d73081e8b2f449138126fc26b5f3a5c2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:10:03.004920 2660 raft_consensus.cc:3037] T 6fca2e9e79da4360893bf0cc0e1101fd P d73081e8b2f449138126fc26b5f3a5c2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:10:03.005620 2660 raft_consensus.cc:504] T 6fca2e9e79da4360893bf0cc0e1101fd P d73081e8b2f449138126fc26b5f3a5c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61c33320fe9844018afb635f9dee3641" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35639 } } peers { permanent_uuid: "166bbea04fd64826a5aae830ef8bfbae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43727 } } peers { permanent_uuid: "d73081e8b2f449138126fc26b5f3a5c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36989 } }
I20260926 01:10:03.005730 2660 leader_election.cc:288] T 6fca2e9e79da4360893bf0cc0e1101fd P d73081e8b2f449138126fc26b5f3a5c2 [CANDIDATE]: Term 1 election: Requested vote from peers 61c33320fe9844018afb635f9dee3641 (127.0.98.1:35639), 166bbea04fd64826a5aae830ef8bfbae (127.0.98.2:43727)
I20260926 01:10:03.005949 2274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fca2e9e79da4360893bf0cc0e1101fd" candidate_uuid: "d73081e8b2f449138126fc26b5f3a5c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61c33320fe9844018afb635f9dee3641"
I20260926 01:10:03.005951 2414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fca2e9e79da4360893bf0cc0e1101fd" candidate_uuid: "d73081e8b2f449138126fc26b5f3a5c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "166bbea04fd64826a5aae830ef8bfbae"
I20260926 01:10:03.006028 2414 raft_consensus.cc:3037] T 6fca2e9e79da4360893bf0cc0e1101fd P 166bbea04fd64826a5aae830ef8bfbae [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:10:03.006028 2274 raft_consensus.cc:3037] T 6fca2e9e79da4360893bf0cc0e1101fd P 61c33320fe9844018afb635f9dee3641 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:10:03.006728 2414 raft_consensus.cc:2454] T 6fca2e9e79da4360893bf0cc0e1101fd P 166bbea04fd64826a5aae830ef8bfbae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d73081e8b2f449138126fc26b5f3a5c2 in term 1.
I20260926 01:10:03.006743 2274 raft_consensus.cc:2454] T 6fca2e9e79da4360893bf0cc0e1101fd P 61c33320fe9844018afb635f9dee3641 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d73081e8b2f449138126fc26b5f3a5c2 in term 1.
I20260926 01:10:03.006911 2488 leader_election.cc:302] T 6fca2e9e79da4360893bf0cc0e1101fd P d73081e8b2f449138126fc26b5f3a5c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 166bbea04fd64826a5aae830ef8bfbae, d73081e8b2f449138126fc26b5f3a5c2; no voters:
I20260926 01:10:03.006999 2660 raft_consensus.cc:2781] T 6fca2e9e79da4360893bf0cc0e1101fd P d73081e8b2f449138126fc26b5f3a5c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:10:03.007050 2660 raft_consensus.cc:686] T 6fca2e9e79da4360893bf0cc0e1101fd P d73081e8b2f449138126fc26b5f3a5c2 [term 1 LEADER]: Becoming Leader. State: Replica: d73081e8b2f449138126fc26b5f3a5c2, State: Running, Role: LEADER
I20260926 01:10:03.007122 2660 consensus_queue.cc:237] T 6fca2e9e79da4360893bf0cc0e1101fd P d73081e8b2f449138126fc26b5f3a5c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61c33320fe9844018afb635f9dee3641" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35639 } } peers { permanent_uuid: "166bbea04fd64826a5aae830ef8bfbae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43727 } } peers { permanent_uuid: "d73081e8b2f449138126fc26b5f3a5c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36989 } }
I20260926 01:10:03.007721 2115 catalog_manager.cc:5495] T 6fca2e9e79da4360893bf0cc0e1101fd P d73081e8b2f449138126fc26b5f3a5c2 reported cstate change: term changed from 0 to 1, leader changed from <none> to d73081e8b2f449138126fc26b5f3a5c2 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "d73081e8b2f449138126fc26b5f3a5c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61c33320fe9844018afb635f9dee3641" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "166bbea04fd64826a5aae830ef8bfbae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43727 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d73081e8b2f449138126fc26b5f3a5c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36989 } health_report { overall_health: HEALTHY } } }
I20260926 01:10:03.041085 2616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:03.039001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42122 (local address 127.0.98.3:36989)
0926 01:10:03.039076 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:03.039077 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:03.039084 (+ 7us) server_negotiation.cc:407] Connection header received
0926 01:10:03.039103 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:03.039104 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:03.039113 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:03.039179 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:10:03.039268 (+ 89us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:03.039767 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:03.040517 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:03.040764 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:03.040834 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:03.040910 (+ 76us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:10:03.040936 (+ 26us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:10:03.040968 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:03.041008 (+ 40us) server_negotiation.cc:299] Negotiation successful
0926 01:10:03.041022 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 01:10:03.045354 2606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:03.043194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36989 (local address 127.0.98.3:41199)
0926 01:10:03.043251 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:03.043263 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:03.043301 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:03.043421 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:03.043421 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:03.043422 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 01:10:03.043458 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:03.043459 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:03.044043 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:03.044044 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:03.044566 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:03.044567 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:03.044941 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:03.044942 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:03.044985 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:03.044989 (+ 4us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 01:10:03.045193 (+ 204us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 01:10:03.045195 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 01:10:03.045242 (+ 47us) client_negotiation.cc:240] Negotiation successful
0926 01:10:03.045282 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260926 01:10:03.045357 2616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:03.043257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41199 (local address 127.0.98.3:36989)
0926 01:10:03.043324 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:03.043326 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:03.043333 (+ 7us) server_negotiation.cc:407] Connection header received
0926 01:10:03.043354 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:03.043355 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:03.043362 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:03.043414 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:10:03.043502 (+ 88us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:03.043989 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:03.044658 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:03.044847 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:03.044909 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:03.045082 (+ 173us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:10:03.045110 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:10:03.045169 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:03.045279 (+ 110us) server_negotiation.cc:299] Negotiation successful
0926 01:10:03.045296 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 01:10:03.047217 2414 raft_consensus.cc:1261] T 6fca2e9e79da4360893bf0cc0e1101fd P 166bbea04fd64826a5aae830ef8bfbae [term 1 FOLLOWER]: Refusing update from remote peer d73081e8b2f449138126fc26b5f3a5c2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:10:03.047267 2274 raft_consensus.cc:1261] T 6fca2e9e79da4360893bf0cc0e1101fd P 61c33320fe9844018afb635f9dee3641 [term 1 FOLLOWER]: Refusing update from remote peer d73081e8b2f449138126fc26b5f3a5c2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:10:03.047429 2627 consensus_queue.cc:1035] T 6fca2e9e79da4360893bf0cc0e1101fd P d73081e8b2f449138126fc26b5f3a5c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61c33320fe9844018afb635f9dee3641" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:10:03.047538 2642 consensus_queue.cc:1035] T 6fca2e9e79da4360893bf0cc0e1101fd P d73081e8b2f449138126fc26b5f3a5c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "166bbea04fd64826a5aae830ef8bfbae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 01:10:03.052260 2647 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260926 01:10:04.570175 2117 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49706:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:10:04.570279 2117 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49706:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:10:04.571249 2117 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 43da41c6d9f34b998e1c66111660121f: Sending DeleteTablet for 3 replicas of tablet 6fca2e9e79da4360893bf0cc0e1101fd
I20260926 01:10:04.571516 2520 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fca2e9e79da4360893bf0cc0e1101fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:10:04 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42100
I20260926 01:10:04.571522 2254 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fca2e9e79da4360893bf0cc0e1101fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:10:04 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53100
I20260926 01:10:04.571796 2698 tablet_replica.cc:330] T 6fca2e9e79da4360893bf0cc0e1101fd P 61c33320fe9844018afb635f9dee3641: stopping tablet replica
I20260926 01:10:04.571871 2698 raft_consensus.cc:2229] T 6fca2e9e79da4360893bf0cc0e1101fd P 61c33320fe9844018afb635f9dee3641 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:10:04.571911 2698 raft_consensus.cc:2258] T 6fca2e9e79da4360893bf0cc0e1101fd P 61c33320fe9844018afb635f9dee3641 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:10:04.572801 2394 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fca2e9e79da4360893bf0cc0e1101fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:10:04 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38970
I20260926 01:10:04.572988 2700 tablet_replica.cc:330] T 6fca2e9e79da4360893bf0cc0e1101fd P 166bbea04fd64826a5aae830ef8bfbae: stopping tablet replica
I20260926 01:10:04.573063 2700 raft_consensus.cc:2229] T 6fca2e9e79da4360893bf0cc0e1101fd P 166bbea04fd64826a5aae830ef8bfbae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:10:04.573108 2700 raft_consensus.cc:2258] T 6fca2e9e79da4360893bf0cc0e1101fd P 166bbea04fd64826a5aae830ef8bfbae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:10:04.573386 2700 ts_tablet_manager.cc:1905] T 6fca2e9e79da4360893bf0cc0e1101fd P 166bbea04fd64826a5aae830ef8bfbae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:10:04.574061 2698 ts_tablet_manager.cc:1905] T 6fca2e9e79da4360893bf0cc0e1101fd P 61c33320fe9844018afb635f9dee3641: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:10:04.574807 2697 tablet_replica.cc:330] T 6fca2e9e79da4360893bf0cc0e1101fd P d73081e8b2f449138126fc26b5f3a5c2: stopping tablet replica
I20260926 01:10:04.574910 2697 raft_consensus.cc:2229] T 6fca2e9e79da4360893bf0cc0e1101fd P d73081e8b2f449138126fc26b5f3a5c2 [term 1 LEADER]: Raft consensus shutting down.
I20260926 01:10:04.575042 2697 raft_consensus.cc:2258] T 6fca2e9e79da4360893bf0cc0e1101fd P d73081e8b2f449138126fc26b5f3a5c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:10:04.575217 2700 ts_tablet_manager.cc:1918] T 6fca2e9e79da4360893bf0cc0e1101fd P 166bbea04fd64826a5aae830ef8bfbae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 01:10:04.575307 2700 log.cc:1196] T 6fca2e9e79da4360893bf0cc0e1101fd P 166bbea04fd64826a5aae830ef8bfbae: Deleting WAL directory at /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384993574762-392-0/minicluster-data/ts-1/wal/wals/6fca2e9e79da4360893bf0cc0e1101fd
I20260926 01:10:04.575326 2697 ts_tablet_manager.cc:1905] T 6fca2e9e79da4360893bf0cc0e1101fd P d73081e8b2f449138126fc26b5f3a5c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:10:04.575551 2700 ts_tablet_manager.cc:1939] T 6fca2e9e79da4360893bf0cc0e1101fd P 166bbea04fd64826a5aae830ef8bfbae: Deleting consensus metadata
I20260926 01:10:04.575881 2698 ts_tablet_manager.cc:1918] T 6fca2e9e79da4360893bf0cc0e1101fd P 61c33320fe9844018afb635f9dee3641: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 01:10:04.575877 2104 catalog_manager.cc:4841] TS 166bbea04fd64826a5aae830ef8bfbae (127.0.98.2:43727): tablet 6fca2e9e79da4360893bf0cc0e1101fd (table test-table [id=1d2003007f8244c496a059cfcedfde53]) successfully deleted
I20260926 01:10:04.575943 2698 log.cc:1196] T 6fca2e9e79da4360893bf0cc0e1101fd P 61c33320fe9844018afb635f9dee3641: Deleting WAL directory at /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384993574762-392-0/minicluster-data/ts-0/wal/wals/6fca2e9e79da4360893bf0cc0e1101fd
I20260926 01:10:04.576159 2698 ts_tablet_manager.cc:1939] T 6fca2e9e79da4360893bf0cc0e1101fd P 61c33320fe9844018afb635f9dee3641: Deleting consensus metadata
I20260926 01:10:04.576620 2104 catalog_manager.cc:4841] TS 61c33320fe9844018afb635f9dee3641 (127.0.98.1:35639): tablet 6fca2e9e79da4360893bf0cc0e1101fd (table test-table [id=1d2003007f8244c496a059cfcedfde53]) successfully deleted
I20260926 01:10:04.576964 2697 ts_tablet_manager.cc:1918] T 6fca2e9e79da4360893bf0cc0e1101fd P d73081e8b2f449138126fc26b5f3a5c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 01:10:04.577015 2697 log.cc:1196] T 6fca2e9e79da4360893bf0cc0e1101fd P d73081e8b2f449138126fc26b5f3a5c2: Deleting WAL directory at /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384993574762-392-0/minicluster-data/ts-2/wal/wals/6fca2e9e79da4360893bf0cc0e1101fd
I20260926 01:10:04.577199 2697 ts_tablet_manager.cc:1939] T 6fca2e9e79da4360893bf0cc0e1101fd P d73081e8b2f449138126fc26b5f3a5c2: Deleting consensus metadata
I20260926 01:10:04.577495 2103 catalog_manager.cc:4841] TS d73081e8b2f449138126fc26b5f3a5c2 (127.0.98.3:36989): tablet 6fca2e9e79da4360893bf0cc0e1101fd (table test-table [id=1d2003007f8244c496a059cfcedfde53]) successfully deleted
Sep 26 01:10:04 dist-test-slave-7b22 krb5kdc[2072](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385004, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 01:10:04 dist-test-slave-7b22 krb5kdc[2072](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385004, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:10:04.604509 2707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:04.587667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49728 (local address 127.0.98.62:43913)
0926 01:10:04.587840 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:04.587841 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:04.587854 (+ 13us) server_negotiation.cc:407] Connection header received
0926 01:10:04.587928 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:04.587929 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:04.587939 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:04.587994 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:04.588717 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:04.589252 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:04.590126 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:04.590334 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:04.590398 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:04.601650 (+ 11252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:10:04.601652 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:10:04.601653 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:10:04.601663 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:10:04.603120 (+ 1457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:04.603544 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:04.603545 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:04.603546 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:04.603601 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:04.603941 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:04.603943 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:04.603943 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:04.604089 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:10:04.604184 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:04.604347 (+ 163us) server_negotiation.cc:299] Negotiation successful
0926 01:10:04.604431 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":80,"threads_started":1}
I20260926 01:10:04.606216 2117 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49728:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:10:04.606393 2117 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:10:04.607733 2394 tablet_service.cc:1467] Processing CreateTablet for tablet e1aaf69eda924f959c5fc65cc232db88 (DEFAULT_TABLE table=test-table [id=a3272b0a158b448080e83687b30c123e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:10:04.607734 2254 tablet_service.cc:1467] Processing CreateTablet for tablet e1aaf69eda924f959c5fc65cc232db88 (DEFAULT_TABLE table=test-table [id=a3272b0a158b448080e83687b30c123e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:10:04.607877 2394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1aaf69eda924f959c5fc65cc232db88. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:10:04.607880 2254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1aaf69eda924f959c5fc65cc232db88. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:10:04.609023 2709 tablet_bootstrap.cc:492] T e1aaf69eda924f959c5fc65cc232db88 P 166bbea04fd64826a5aae830ef8bfbae: Bootstrap starting.
I20260926 01:10:04.609036 2520 tablet_service.cc:1467] Processing CreateTablet for tablet e1aaf69eda924f959c5fc65cc232db88 (DEFAULT_TABLE table=test-table [id=a3272b0a158b448080e83687b30c123e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:10:04.609050 2708 tablet_bootstrap.cc:492] T e1aaf69eda924f959c5fc65cc232db88 P 61c33320fe9844018afb635f9dee3641: Bootstrap starting.
I20260926 01:10:04.609149 2520 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1aaf69eda924f959c5fc65cc232db88. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:10:04.609426 2708 tablet_bootstrap.cc:654] T e1aaf69eda924f959c5fc65cc232db88 P 61c33320fe9844018afb635f9dee3641: Neither blocks nor log segments found. Creating new log.
I20260926 01:10:04.609480 2709 tablet_bootstrap.cc:654] T e1aaf69eda924f959c5fc65cc232db88 P 166bbea04fd64826a5aae830ef8bfbae: Neither blocks nor log segments found. Creating new log.
I20260926 01:10:04.609994 2708 tablet_bootstrap.cc:492] T e1aaf69eda924f959c5fc65cc232db88 P 61c33320fe9844018afb635f9dee3641: No bootstrap required, opened a new log
I20260926 01:10:04.610040 2708 ts_tablet_manager.cc:1397] T e1aaf69eda924f959c5fc65cc232db88 P 61c33320fe9844018afb635f9dee3641: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 01:10:04.610046 2709 tablet_bootstrap.cc:492] T e1aaf69eda924f959c5fc65cc232db88 P 166bbea04fd64826a5aae830ef8bfbae: No bootstrap required, opened a new log
I20260926 01:10:04.610082 2709 ts_tablet_manager.cc:1397] T e1aaf69eda924f959c5fc65cc232db88 P 166bbea04fd64826a5aae830ef8bfbae: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 01:10:04.610173 2709 raft_consensus.cc:348] T e1aaf69eda924f959c5fc65cc232db88 P 166bbea04fd64826a5aae830ef8bfbae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61c33320fe9844018afb635f9dee3641" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35639 } } peers { permanent_uuid: "166bbea04fd64826a5aae830ef8bfbae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43727 } } peers { permanent_uuid: "d73081e8b2f449138126fc26b5f3a5c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36989 } }
I20260926 01:10:04.610162 2708 raft_consensus.cc:348] T e1aaf69eda924f959c5fc65cc232db88 P 61c33320fe9844018afb635f9dee3641 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61c33320fe9844018afb635f9dee3641" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35639 } } peers { permanent_uuid: "166bbea04fd64826a5aae830ef8bfbae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43727 } } peers { permanent_uuid: "d73081e8b2f449138126fc26b5f3a5c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36989 } }
I20260926 01:10:04.610224 2709 raft_consensus.cc:374] T e1aaf69eda924f959c5fc65cc232db88 P 166bbea04fd64826a5aae830ef8bfbae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:10:04.610229 2708 raft_consensus.cc:374] T e1aaf69eda924f959c5fc65cc232db88 P 61c33320fe9844018afb635f9dee3641 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:10:04.610234 2709 raft_consensus.cc:729] T e1aaf69eda924f959c5fc65cc232db88 P 166bbea04fd64826a5aae830ef8bfbae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 166bbea04fd64826a5aae830ef8bfbae, State: Initialized, Role: FOLLOWER
I20260926 01:10:04.610247 2708 raft_consensus.cc:729] T e1aaf69eda924f959c5fc65cc232db88 P 61c33320fe9844018afb635f9dee3641 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61c33320fe9844018afb635f9dee3641, State: Initialized, Role: FOLLOWER
I20260926 01:10:04.610275 2709 consensus_queue.cc:260] T e1aaf69eda924f959c5fc65cc232db88 P 166bbea04fd64826a5aae830ef8bfbae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61c33320fe9844018afb635f9dee3641" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35639 } } peers { permanent_uuid: "166bbea04fd64826a5aae830ef8bfbae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43727 } } peers { permanent_uuid: "d73081e8b2f449138126fc26b5f3a5c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36989 } }
I20260926 01:10:04.610291 2708 consensus_queue.cc:260] T e1aaf69eda924f959c5fc65cc232db88 P 61c33320fe9844018afb635f9dee3641 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61c33320fe9844018afb635f9dee3641" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35639 } } peers { permanent_uuid: "166bbea04fd64826a5aae830ef8bfbae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43727 } } peers { permanent_uuid: "d73081e8b2f449138126fc26b5f3a5c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36989 } }
I20260926 01:10:04.610428 2709 ts_tablet_manager.cc:1428] T e1aaf69eda924f959c5fc65cc232db88 P 166bbea04fd64826a5aae830ef8bfbae: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:04.610442 2708 ts_tablet_manager.cc:1428] T e1aaf69eda924f959c5fc65cc232db88 P 61c33320fe9844018afb635f9dee3641: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 01:10:04.610580 2714 tablet_bootstrap.cc:492] T e1aaf69eda924f959c5fc65cc232db88 P d73081e8b2f449138126fc26b5f3a5c2: Bootstrap starting.
I20260926 01:10:04.610940 2714 tablet_bootstrap.cc:654] T e1aaf69eda924f959c5fc65cc232db88 P d73081e8b2f449138126fc26b5f3a5c2: Neither blocks nor log segments found. Creating new log.
I20260926 01:10:04.611604 2714 tablet_bootstrap.cc:492] T e1aaf69eda924f959c5fc65cc232db88 P d73081e8b2f449138126fc26b5f3a5c2: No bootstrap required, opened a new log
I20260926 01:10:04.611693 2714 ts_tablet_manager.cc:1397] T e1aaf69eda924f959c5fc65cc232db88 P d73081e8b2f449138126fc26b5f3a5c2: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 01:10:04.611826 2714 raft_consensus.cc:348] T e1aaf69eda924f959c5fc65cc232db88 P d73081e8b2f449138126fc26b5f3a5c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61c33320fe9844018afb635f9dee3641" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35639 } } peers { permanent_uuid: "166bbea04fd64826a5aae830ef8bfbae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43727 } } peers { permanent_uuid: "d73081e8b2f449138126fc26b5f3a5c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36989 } }
I20260926 01:10:04.611891 2714 raft_consensus.cc:374] T e1aaf69eda924f959c5fc65cc232db88 P d73081e8b2f449138126fc26b5f3a5c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:10:04.611903 2714 raft_consensus.cc:729] T e1aaf69eda924f959c5fc65cc232db88 P d73081e8b2f449138126fc26b5f3a5c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d73081e8b2f449138126fc26b5f3a5c2, State: Initialized, Role: FOLLOWER
I20260926 01:10:04.611938 2714 consensus_queue.cc:260] T e1aaf69eda924f959c5fc65cc232db88 P d73081e8b2f449138126fc26b5f3a5c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61c33320fe9844018afb635f9dee3641" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35639 } } peers { permanent_uuid: "166bbea04fd64826a5aae830ef8bfbae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43727 } } peers { permanent_uuid: "d73081e8b2f449138126fc26b5f3a5c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36989 } }
I20260926 01:10:04.612030 2714 ts_tablet_manager.cc:1428] T e1aaf69eda924f959c5fc65cc232db88 P d73081e8b2f449138126fc26b5f3a5c2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:04.719224 2713 raft_consensus.cc:482] T e1aaf69eda924f959c5fc65cc232db88 P 61c33320fe9844018afb635f9dee3641 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:10:04.719329 2713 raft_consensus.cc:504] T e1aaf69eda924f959c5fc65cc232db88 P 61c33320fe9844018afb635f9dee3641 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61c33320fe9844018afb635f9dee3641" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35639 } } peers { permanent_uuid: "166bbea04fd64826a5aae830ef8bfbae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43727 } } peers { permanent_uuid: "d73081e8b2f449138126fc26b5f3a5c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36989 } }
I20260926 01:10:04.719657 2713 leader_election.cc:288] T e1aaf69eda924f959c5fc65cc232db88 P 61c33320fe9844018afb635f9dee3641 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 166bbea04fd64826a5aae830ef8bfbae (127.0.98.2:43727), d73081e8b2f449138126fc26b5f3a5c2 (127.0.98.3:36989)
I20260926 01:10:04.723702 2716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:04.719930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57813 (local address 127.0.98.2:43727)
0926 01:10:04.720166 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:04.720168 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:04.720177 (+ 9us) server_negotiation.cc:407] Connection header received
0926 01:10:04.720229 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:04.720230 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:04.720240 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:04.720308 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:10:04.720524 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:04.721308 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:04.722836 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:04.723320 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:04.723394 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:04.723408 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:04.723581 (+ 173us) server_negotiation.cc:299] Negotiation successful
0926 01:10:04.723624 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":88,"threads_started":1}
I20260926 01:10:04.723706 2717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:04.719879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43727 (local address 127.0.98.1:57813)
0926 01:10:04.720079 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:04.720091 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:04.720177 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:04.720316 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:04.720316 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:04.720319 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:10:04.720411 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:04.720413 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:04.721425 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:04.721426 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:04.722741 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:04.722745 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:04.723433 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:04.723435 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:04.723502 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:04.723506 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 01:10:04.723562 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 01:10:04.723599 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"mutex_wait_us":59,"thread_start_us":144,"threads_started":1}
I20260926 01:10:04.723964 2718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:04.719893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36989 (local address 127.0.98.1:51903)
0926 01:10:04.720082 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:04.720091 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:04.720152 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:04.720316 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:04.720316 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:04.720320 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:10:04.720411 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:04.720413 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:04.721408 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:04.721409 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:04.722985 (+ 1576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:04.722987 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:04.723720 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:04.723722 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:04.723786 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:04.723790 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 01:10:04.723845 (+ 55us) client_negotiation.cc:240] Negotiation successful
0926 01:10:04.723887 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"mutex_wait_us":43,"thread_start_us":121,"threads_started":1}
I20260926 01:10:04.723948 2413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1aaf69eda924f959c5fc65cc232db88" candidate_uuid: "61c33320fe9844018afb635f9dee3641" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "166bbea04fd64826a5aae830ef8bfbae" is_pre_election: true
I20260926 01:10:04.724030 2413 raft_consensus.cc:2454] T e1aaf69eda924f959c5fc65cc232db88 P 166bbea04fd64826a5aae830ef8bfbae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61c33320fe9844018afb635f9dee3641 in term 0.
I20260926 01:10:04.724057 2719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:04.719993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51903 (local address 127.0.98.3:36989)
0926 01:10:04.720139 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:04.720141 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:04.720153 (+ 12us) server_negotiation.cc:407] Connection header received
0926 01:10:04.720209 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:04.720210 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:04.720221 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:04.720313 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:10:04.720524 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:04.721304 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:04.723080 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:04.723592 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:04.723657 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:04.723669 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:04.723931 (+ 262us) server_negotiation.cc:299] Negotiation successful
0926 01:10:04.723987 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":92,"threads_started":1}
I20260926 01:10:04.724232 2208 leader_election.cc:302] T e1aaf69eda924f959c5fc65cc232db88 P 61c33320fe9844018afb635f9dee3641 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 166bbea04fd64826a5aae830ef8bfbae, 61c33320fe9844018afb635f9dee3641; no voters:
I20260926 01:10:04.724368 2713 raft_consensus.cc:2781] T e1aaf69eda924f959c5fc65cc232db88 P 61c33320fe9844018afb635f9dee3641 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:10:04.724342 2554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1aaf69eda924f959c5fc65cc232db88" candidate_uuid: "61c33320fe9844018afb635f9dee3641" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73081e8b2f449138126fc26b5f3a5c2" is_pre_election: true
I20260926 01:10:04.724406 2713 raft_consensus.cc:482] T e1aaf69eda924f959c5fc65cc232db88 P 61c33320fe9844018afb635f9dee3641 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:10:04.724426 2713 raft_consensus.cc:3037] T e1aaf69eda924f959c5fc65cc232db88 P 61c33320fe9844018afb635f9dee3641 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:10:04.724421 2554 raft_consensus.cc:2454] T e1aaf69eda924f959c5fc65cc232db88 P d73081e8b2f449138126fc26b5f3a5c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61c33320fe9844018afb635f9dee3641 in term 0.
I20260926 01:10:04.725157 2713 raft_consensus.cc:504] T e1aaf69eda924f959c5fc65cc232db88 P 61c33320fe9844018afb635f9dee3641 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61c33320fe9844018afb635f9dee3641" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35639 } } peers { permanent_uuid: "166bbea04fd64826a5aae830ef8bfbae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43727 } } peers { permanent_uuid: "d73081e8b2f449138126fc26b5f3a5c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36989 } }
I20260926 01:10:04.725337 2713 leader_election.cc:288] T e1aaf69eda924f959c5fc65cc232db88 P 61c33320fe9844018afb635f9dee3641 [CANDIDATE]: Term 1 election: Requested vote from peers 166bbea04fd64826a5aae830ef8bfbae (127.0.98.2:43727), d73081e8b2f449138126fc26b5f3a5c2 (127.0.98.3:36989)
I20260926 01:10:04.725498 2554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1aaf69eda924f959c5fc65cc232db88" candidate_uuid: "61c33320fe9844018afb635f9dee3641" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73081e8b2f449138126fc26b5f3a5c2"
I20260926 01:10:04.725507 2413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1aaf69eda924f959c5fc65cc232db88" candidate_uuid: "61c33320fe9844018afb635f9dee3641" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "166bbea04fd64826a5aae830ef8bfbae"
I20260926 01:10:04.725572 2554 raft_consensus.cc:3037] T e1aaf69eda924f959c5fc65cc232db88 P d73081e8b2f449138126fc26b5f3a5c2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:10:04.725576 2413 raft_consensus.cc:3037] T e1aaf69eda924f959c5fc65cc232db88 P 166bbea04fd64826a5aae830ef8bfbae [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:10:04.726269 2554 raft_consensus.cc:2454] T e1aaf69eda924f959c5fc65cc232db88 P d73081e8b2f449138126fc26b5f3a5c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61c33320fe9844018afb635f9dee3641 in term 1.
I20260926 01:10:04.726279 2413 raft_consensus.cc:2454] T e1aaf69eda924f959c5fc65cc232db88 P 166bbea04fd64826a5aae830ef8bfbae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61c33320fe9844018afb635f9dee3641 in term 1.
I20260926 01:10:04.726401 2208 leader_election.cc:302] T e1aaf69eda924f959c5fc65cc232db88 P 61c33320fe9844018afb635f9dee3641 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 166bbea04fd64826a5aae830ef8bfbae, 61c33320fe9844018afb635f9dee3641; no voters:
I20260926 01:10:04.726500 2713 raft_consensus.cc:2781] T e1aaf69eda924f959c5fc65cc232db88 P 61c33320fe9844018afb635f9dee3641 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:10:04.726624 2713 raft_consensus.cc:686] T e1aaf69eda924f959c5fc65cc232db88 P 61c33320fe9844018afb635f9dee3641 [term 1 LEADER]: Becoming Leader. State: Replica: 61c33320fe9844018afb635f9dee3641, State: Running, Role: LEADER
I20260926 01:10:04.726682 2713 consensus_queue.cc:237] T e1aaf69eda924f959c5fc65cc232db88 P 61c33320fe9844018afb635f9dee3641 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61c33320fe9844018afb635f9dee3641" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35639 } } peers { permanent_uuid: "166bbea04fd64826a5aae830ef8bfbae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43727 } } peers { permanent_uuid: "d73081e8b2f449138126fc26b5f3a5c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36989 } }
I20260926 01:10:04.727370 2117 catalog_manager.cc:5495] T e1aaf69eda924f959c5fc65cc232db88 P 61c33320fe9844018afb635f9dee3641 reported cstate change: term changed from 0 to 1, leader changed from <none> to 61c33320fe9844018afb635f9dee3641 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "61c33320fe9844018afb635f9dee3641" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61c33320fe9844018afb635f9dee3641" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "166bbea04fd64826a5aae830ef8bfbae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43727 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d73081e8b2f449138126fc26b5f3a5c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36989 } health_report { overall_health: UNKNOWN } } }
I20260926 01:10:04.761406 2723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:04.759023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53108 (local address 127.0.98.1:35639)
0926 01:10:04.759207 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:04.759209 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:04.759216 (+ 7us) server_negotiation.cc:407] Connection header received
0926 01:10:04.759258 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:04.759258 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:04.759285 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:04.759362 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:10:04.759488 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:04.760018 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:04.760746 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:04.760948 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:04.761005 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:04.761104 (+ 99us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:10:04.761210 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:10:04.761261 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:04.761293 (+ 32us) server_negotiation.cc:299] Negotiation successful
0926 01:10:04.761310 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":98,"threads_started":1}
I20260926 01:10:04.766165 2719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:04.763822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37853 (local address 127.0.98.3:36989)
0926 01:10:04.763876 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:04.763878 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:04.763908 (+ 30us) server_negotiation.cc:407] Connection header received
0926 01:10:04.763982 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:04.763983 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:04.763994 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:04.764045 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:10:04.764187 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:04.764659 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:04.765406 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:04.765648 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:04.765702 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:04.765930 (+ 228us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:10:04.765965 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:10:04.766008 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:04.766064 (+ 56us) server_negotiation.cc:299] Negotiation successful
0926 01:10:04.766085 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 01:10:04.766194 2726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:04.763641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36989 (local address 127.0.98.1:37853)
0926 01:10:04.763848 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:04.763860 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:04.763924 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:04.764049 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:04.764050 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:04.764051 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 01:10:04.764111 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:04.764113 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:04.764726 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:04.764727 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:04.765330 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:04.765331 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:04.765708 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:04.765709 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:04.765799 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:04.765858 (+ 59us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 01:10:04.766021 (+ 163us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 01:10:04.766022 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 01:10:04.766061 (+ 39us) client_negotiation.cc:240] Negotiation successful
0926 01:10:04.766105 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":102,"threads_started":1}
I20260926 01:10:04.768517 2413 raft_consensus.cc:1261] T e1aaf69eda924f959c5fc65cc232db88 P 166bbea04fd64826a5aae830ef8bfbae [term 1 FOLLOWER]: Refusing update from remote peer 61c33320fe9844018afb635f9dee3641: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:10:04.768554 2554 raft_consensus.cc:1261] T e1aaf69eda924f959c5fc65cc232db88 P d73081e8b2f449138126fc26b5f3a5c2 [term 1 FOLLOWER]: Refusing update from remote peer 61c33320fe9844018afb635f9dee3641: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:10:04.768784 2720 consensus_queue.cc:1035] T e1aaf69eda924f959c5fc65cc232db88 P 61c33320fe9844018afb635f9dee3641 [LEADER]: Connected to new peer: Peer: permanent_uuid: "166bbea04fd64826a5aae830ef8bfbae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:10:04.768915 2713 consensus_queue.cc:1035] T e1aaf69eda924f959c5fc65cc232db88 P 61c33320fe9844018afb635f9dee3641 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d73081e8b2f449138126fc26b5f3a5c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:10:04.775513 2723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:04.773171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:32865 (local address 127.0.98.1:35639)
0926 01:10:04.773262 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:04.773264 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:04.773365 (+ 101us) server_negotiation.cc:407] Connection header received
0926 01:10:04.773420 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:04.773421 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:04.773429 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:04.773462 (+ 33us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:10:04.773542 (+ 80us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:04.774089 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:04.774841 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:04.775108 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:04.775192 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:04.775338 (+ 146us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:10:04.775367 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:10:04.775402 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:04.775441 (+ 39us) server_negotiation.cc:299] Negotiation successful
0926 01:10:04.775458 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 01:10:04.775537 2730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:04.773114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35639 (local address 127.0.98.3:32865)
0926 01:10:04.773315 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:04.773326 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:04.773375 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:04.773464 (+ 89us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:04.773465 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:04.773466 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 01:10:04.773508 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:04.773509 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:04.774143 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:04.774144 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:04.774759 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:04.774761 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:04.775204 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:04.775205 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:04.775265 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:04.775272 (+ 7us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 01:10:04.775405 (+ 133us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 01:10:04.775407 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 01:10:04.775438 (+ 31us) client_negotiation.cc:240] Negotiation successful
0926 01:10:04.775477 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":86,"threads_started":1}
I20260926 01:10:04.781596 2116 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49728:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:10:04.781728 2116 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49728:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:10:04.782421 2116 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 43da41c6d9f34b998e1c66111660121f: Sending DeleteTablet for 3 replicas of tablet e1aaf69eda924f959c5fc65cc232db88
I20260926 01:10:04.782719 2394 tablet_service.cc:1514] Processing DeleteTablet for tablet e1aaf69eda924f959c5fc65cc232db88 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:10:04 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38970
I20260926 01:10:04.782764 2254 tablet_service.cc:1514] Processing DeleteTablet for tablet e1aaf69eda924f959c5fc65cc232db88 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:10:04 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53100
I20260926 01:10:04.782775 2520 tablet_service.cc:1514] Processing DeleteTablet for tablet e1aaf69eda924f959c5fc65cc232db88 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:10:04 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42100
I20260926 01:10:04.782821 2700 tablet_replica.cc:330] T e1aaf69eda924f959c5fc65cc232db88 P 166bbea04fd64826a5aae830ef8bfbae: stopping tablet replica
I20260926 01:10:04.782857 2698 tablet_replica.cc:330] T e1aaf69eda924f959c5fc65cc232db88 P 61c33320fe9844018afb635f9dee3641: stopping tablet replica
I20260926 01:10:04.782871 2700 raft_consensus.cc:2229] T e1aaf69eda924f959c5fc65cc232db88 P 166bbea04fd64826a5aae830ef8bfbae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:10:04.782899 2698 raft_consensus.cc:2229] T e1aaf69eda924f959c5fc65cc232db88 P 61c33320fe9844018afb635f9dee3641 [term 1 LEADER]: Raft consensus shutting down.
I20260926 01:10:04.782912 2700 raft_consensus.cc:2258] T e1aaf69eda924f959c5fc65cc232db88 P 166bbea04fd64826a5aae830ef8bfbae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:10:04.782989 2698 raft_consensus.cc:2258] T e1aaf69eda924f959c5fc65cc232db88 P 61c33320fe9844018afb635f9dee3641 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:10:04.783067 2697 tablet_replica.cc:330] T e1aaf69eda924f959c5fc65cc232db88 P d73081e8b2f449138126fc26b5f3a5c2: stopping tablet replica
I20260926 01:10:04.783119 2697 raft_consensus.cc:2229] T e1aaf69eda924f959c5fc65cc232db88 P d73081e8b2f449138126fc26b5f3a5c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:10:04.783151 2697 raft_consensus.cc:2258] T e1aaf69eda924f959c5fc65cc232db88 P d73081e8b2f449138126fc26b5f3a5c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:10:04.783224 2116 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49728:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 01:10:04.783340 2700 ts_tablet_manager.cc:1905] T e1aaf69eda924f959c5fc65cc232db88 P 166bbea04fd64826a5aae830ef8bfbae: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 01:10:04.783365 2116 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:10:04.783535 2698 ts_tablet_manager.cc:1905] T e1aaf69eda924f959c5fc65cc232db88 P 61c33320fe9844018afb635f9dee3641: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:10:04.784399 2394 tablet_service.cc:1467] Processing CreateTablet for tablet da53a21a08d14e2b8e0c5178078714f1 (DEFAULT_TABLE table=test-table [id=c90ecd42675a4eafb797d1debc38d230]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:10:04.784490 2394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da53a21a08d14e2b8e0c5178078714f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:10:04.785085 2254 tablet_service.cc:1467] Processing CreateTablet for tablet da53a21a08d14e2b8e0c5178078714f1 (DEFAULT_TABLE table=test-table [id=c90ecd42675a4eafb797d1debc38d230]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:10:04.785185 2254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da53a21a08d14e2b8e0c5178078714f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:10:04.785194 2698 ts_tablet_manager.cc:1918] T e1aaf69eda924f959c5fc65cc232db88 P 61c33320fe9844018afb635f9dee3641: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 01:10:04.785246 2698 log.cc:1196] T e1aaf69eda924f959c5fc65cc232db88 P 61c33320fe9844018afb635f9dee3641: Deleting WAL directory at /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384993574762-392-0/minicluster-data/ts-0/wal/wals/e1aaf69eda924f959c5fc65cc232db88
I20260926 01:10:04.785257 2700 ts_tablet_manager.cc:1918] T e1aaf69eda924f959c5fc65cc232db88 P 166bbea04fd64826a5aae830ef8bfbae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 01:10:04.785300 2700 log.cc:1196] T e1aaf69eda924f959c5fc65cc232db88 P 166bbea04fd64826a5aae830ef8bfbae: Deleting WAL directory at /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384993574762-392-0/minicluster-data/ts-1/wal/wals/e1aaf69eda924f959c5fc65cc232db88
I20260926 01:10:04.785460 2698 ts_tablet_manager.cc:1939] T e1aaf69eda924f959c5fc65cc232db88 P 61c33320fe9844018afb635f9dee3641: Deleting consensus metadata
I20260926 01:10:04.785518 2700 ts_tablet_manager.cc:1939] T e1aaf69eda924f959c5fc65cc232db88 P 166bbea04fd64826a5aae830ef8bfbae: Deleting consensus metadata
I20260926 01:10:04.785907 2697 ts_tablet_manager.cc:1905] T e1aaf69eda924f959c5fc65cc232db88 P d73081e8b2f449138126fc26b5f3a5c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:10:04.786331 2708 tablet_bootstrap.cc:492] T da53a21a08d14e2b8e0c5178078714f1 P 61c33320fe9844018afb635f9dee3641: Bootstrap starting.
I20260926 01:10:04.786412 2104 catalog_manager.cc:4841] TS 166bbea04fd64826a5aae830ef8bfbae (127.0.98.2:43727): tablet e1aaf69eda924f959c5fc65cc232db88 (table test-table [id=a3272b0a158b448080e83687b30c123e]) successfully deleted
I20260926 01:10:04.786550 2104 catalog_manager.cc:4841] TS 61c33320fe9844018afb635f9dee3641 (127.0.98.1:35639): tablet e1aaf69eda924f959c5fc65cc232db88 (table test-table [id=a3272b0a158b448080e83687b30c123e]) successfully deleted
I20260926 01:10:04.786710 2708 tablet_bootstrap.cc:654] T da53a21a08d14e2b8e0c5178078714f1 P 61c33320fe9844018afb635f9dee3641: Neither blocks nor log segments found. Creating new log.
I20260926 01:10:04.786931 2520 tablet_service.cc:1467] Processing CreateTablet for tablet da53a21a08d14e2b8e0c5178078714f1 (DEFAULT_TABLE table=test-table [id=c90ecd42675a4eafb797d1debc38d230]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:10:04.787014 2520 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da53a21a08d14e2b8e0c5178078714f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:10:04.787320 2708 tablet_bootstrap.cc:492] T da53a21a08d14e2b8e0c5178078714f1 P 61c33320fe9844018afb635f9dee3641: No bootstrap required, opened a new log
I20260926 01:10:04.787361 2708 ts_tablet_manager.cc:1397] T da53a21a08d14e2b8e0c5178078714f1 P 61c33320fe9844018afb635f9dee3641: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 01:10:04.787434 2697 ts_tablet_manager.cc:1918] T e1aaf69eda924f959c5fc65cc232db88 P d73081e8b2f449138126fc26b5f3a5c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 01:10:04.787480 2697 log.cc:1196] T e1aaf69eda924f959c5fc65cc232db88 P d73081e8b2f449138126fc26b5f3a5c2: Deleting WAL directory at /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384993574762-392-0/minicluster-data/ts-2/wal/wals/e1aaf69eda924f959c5fc65cc232db88
I20260926 01:10:04.787526 2708 raft_consensus.cc:348] T da53a21a08d14e2b8e0c5178078714f1 P 61c33320fe9844018afb635f9dee3641 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61c33320fe9844018afb635f9dee3641" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35639 } } peers { permanent_uuid: "166bbea04fd64826a5aae830ef8bfbae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43727 } } peers { permanent_uuid: "d73081e8b2f449138126fc26b5f3a5c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36989 } }
I20260926 01:10:04.787591 2708 raft_consensus.cc:374] T da53a21a08d14e2b8e0c5178078714f1 P 61c33320fe9844018afb635f9dee3641 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:10:04.787611 2708 raft_consensus.cc:729] T da53a21a08d14e2b8e0c5178078714f1 P 61c33320fe9844018afb635f9dee3641 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61c33320fe9844018afb635f9dee3641, State: Initialized, Role: FOLLOWER
I20260926 01:10:04.787659 2697 ts_tablet_manager.cc:1939] T e1aaf69eda924f959c5fc65cc232db88 P d73081e8b2f449138126fc26b5f3a5c2: Deleting consensus metadata
I20260926 01:10:04.787650 2708 consensus_queue.cc:260] T da53a21a08d14e2b8e0c5178078714f1 P 61c33320fe9844018afb635f9dee3641 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61c33320fe9844018afb635f9dee3641" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35639 } } peers { permanent_uuid: "166bbea04fd64826a5aae830ef8bfbae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43727 } } peers { permanent_uuid: "d73081e8b2f449138126fc26b5f3a5c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36989 } }
I20260926 01:10:04.787760 2708 ts_tablet_manager.cc:1428] T da53a21a08d14e2b8e0c5178078714f1 P 61c33320fe9844018afb635f9dee3641: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 01:10:04.788159 2714 tablet_bootstrap.cc:492] T da53a21a08d14e2b8e0c5178078714f1 P d73081e8b2f449138126fc26b5f3a5c2: Bootstrap starting.
I20260926 01:10:04.788475 2714 tablet_bootstrap.cc:654] T da53a21a08d14e2b8e0c5178078714f1 P d73081e8b2f449138126fc26b5f3a5c2: Neither blocks nor log segments found. Creating new log.
I20260926 01:10:04.788733 2103 catalog_manager.cc:4841] TS d73081e8b2f449138126fc26b5f3a5c2 (127.0.98.3:36989): tablet e1aaf69eda924f959c5fc65cc232db88 (table test-table [id=a3272b0a158b448080e83687b30c123e]) successfully deleted
I20260926 01:10:04.788970 2714 tablet_bootstrap.cc:492] T da53a21a08d14e2b8e0c5178078714f1 P d73081e8b2f449138126fc26b5f3a5c2: No bootstrap required, opened a new log
I20260926 01:10:04.789016 2714 ts_tablet_manager.cc:1397] T da53a21a08d14e2b8e0c5178078714f1 P d73081e8b2f449138126fc26b5f3a5c2: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 01:10:04.789125 2714 raft_consensus.cc:348] T da53a21a08d14e2b8e0c5178078714f1 P d73081e8b2f449138126fc26b5f3a5c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61c33320fe9844018afb635f9dee3641" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35639 } } peers { permanent_uuid: "166bbea04fd64826a5aae830ef8bfbae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43727 } } peers { permanent_uuid: "d73081e8b2f449138126fc26b5f3a5c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36989 } }
I20260926 01:10:04.789186 2714 raft_consensus.cc:374] T da53a21a08d14e2b8e0c5178078714f1 P d73081e8b2f449138126fc26b5f3a5c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:10:04.789201 2714 raft_consensus.cc:729] T da53a21a08d14e2b8e0c5178078714f1 P d73081e8b2f449138126fc26b5f3a5c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d73081e8b2f449138126fc26b5f3a5c2, State: Initialized, Role: FOLLOWER
I20260926 01:10:04.789235 2714 consensus_queue.cc:260] T da53a21a08d14e2b8e0c5178078714f1 P d73081e8b2f449138126fc26b5f3a5c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61c33320fe9844018afb635f9dee3641" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35639 } } peers { permanent_uuid: "166bbea04fd64826a5aae830ef8bfbae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43727 } } peers { permanent_uuid: "d73081e8b2f449138126fc26b5f3a5c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36989 } }
I20260926 01:10:04.789309 2714 ts_tablet_manager.cc:1428] T da53a21a08d14e2b8e0c5178078714f1 P d73081e8b2f449138126fc26b5f3a5c2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:04.789716 2709 tablet_bootstrap.cc:492] T da53a21a08d14e2b8e0c5178078714f1 P 166bbea04fd64826a5aae830ef8bfbae: Bootstrap starting.
I20260926 01:10:04.790024 2709 tablet_bootstrap.cc:654] T da53a21a08d14e2b8e0c5178078714f1 P 166bbea04fd64826a5aae830ef8bfbae: Neither blocks nor log segments found. Creating new log.
I20260926 01:10:04.790535 2709 tablet_bootstrap.cc:492] T da53a21a08d14e2b8e0c5178078714f1 P 166bbea04fd64826a5aae830ef8bfbae: No bootstrap required, opened a new log
I20260926 01:10:04.790592 2709 ts_tablet_manager.cc:1397] T da53a21a08d14e2b8e0c5178078714f1 P 166bbea04fd64826a5aae830ef8bfbae: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 01:10:04.790712 2709 raft_consensus.cc:348] T da53a21a08d14e2b8e0c5178078714f1 P 166bbea04fd64826a5aae830ef8bfbae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61c33320fe9844018afb635f9dee3641" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35639 } } peers { permanent_uuid: "166bbea04fd64826a5aae830ef8bfbae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43727 } } peers { permanent_uuid: "d73081e8b2f449138126fc26b5f3a5c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36989 } }
I20260926 01:10:04.790759 2709 raft_consensus.cc:374] T da53a21a08d14e2b8e0c5178078714f1 P 166bbea04fd64826a5aae830ef8bfbae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:10:04.790771 2709 raft_consensus.cc:729] T da53a21a08d14e2b8e0c5178078714f1 P 166bbea04fd64826a5aae830ef8bfbae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 166bbea04fd64826a5aae830ef8bfbae, State: Initialized, Role: FOLLOWER
I20260926 01:10:04.790800 2709 consensus_queue.cc:260] T da53a21a08d14e2b8e0c5178078714f1 P 166bbea04fd64826a5aae830ef8bfbae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61c33320fe9844018afb635f9dee3641" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35639 } } peers { permanent_uuid: "166bbea04fd64826a5aae830ef8bfbae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43727 } } peers { permanent_uuid: "d73081e8b2f449138126fc26b5f3a5c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36989 } }
I20260926 01:10:04.790863 2709 ts_tablet_manager.cc:1428] T da53a21a08d14e2b8e0c5178078714f1 P 166bbea04fd64826a5aae830ef8bfbae: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:04.918351 2712 raft_consensus.cc:482] T da53a21a08d14e2b8e0c5178078714f1 P 166bbea04fd64826a5aae830ef8bfbae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:10:04.918457 2712 raft_consensus.cc:504] T da53a21a08d14e2b8e0c5178078714f1 P 166bbea04fd64826a5aae830ef8bfbae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61c33320fe9844018afb635f9dee3641" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35639 } } peers { permanent_uuid: "166bbea04fd64826a5aae830ef8bfbae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43727 } } peers { permanent_uuid: "d73081e8b2f449138126fc26b5f3a5c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36989 } }
I20260926 01:10:04.918644 2712 leader_election.cc:288] T da53a21a08d14e2b8e0c5178078714f1 P 166bbea04fd64826a5aae830ef8bfbae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 61c33320fe9844018afb635f9dee3641 (127.0.98.1:35639), d73081e8b2f449138126fc26b5f3a5c2 (127.0.98.3:36989)
I20260926 01:10:04.918895 2554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da53a21a08d14e2b8e0c5178078714f1" candidate_uuid: "166bbea04fd64826a5aae830ef8bfbae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73081e8b2f449138126fc26b5f3a5c2" is_pre_election: true
I20260926 01:10:04.918898 2274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da53a21a08d14e2b8e0c5178078714f1" candidate_uuid: "166bbea04fd64826a5aae830ef8bfbae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61c33320fe9844018afb635f9dee3641" is_pre_election: true
I20260926 01:10:04.918990 2554 raft_consensus.cc:2454] T da53a21a08d14e2b8e0c5178078714f1 P d73081e8b2f449138126fc26b5f3a5c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 166bbea04fd64826a5aae830ef8bfbae in term 0.
I20260926 01:10:04.918994 2274 raft_consensus.cc:2454] T da53a21a08d14e2b8e0c5178078714f1 P 61c33320fe9844018afb635f9dee3641 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 166bbea04fd64826a5aae830ef8bfbae in term 0.
I20260926 01:10:04.919152 2347 leader_election.cc:302] T da53a21a08d14e2b8e0c5178078714f1 P 166bbea04fd64826a5aae830ef8bfbae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 166bbea04fd64826a5aae830ef8bfbae, d73081e8b2f449138126fc26b5f3a5c2; no voters:
I20260926 01:10:04.919260 2712 raft_consensus.cc:2781] T da53a21a08d14e2b8e0c5178078714f1 P 166bbea04fd64826a5aae830ef8bfbae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:10:04.919314 2712 raft_consensus.cc:482] T da53a21a08d14e2b8e0c5178078714f1 P 166bbea04fd64826a5aae830ef8bfbae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:10:04.919333 2712 raft_consensus.cc:3037] T da53a21a08d14e2b8e0c5178078714f1 P 166bbea04fd64826a5aae830ef8bfbae [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:10:04.920068 2712 raft_consensus.cc:504] T da53a21a08d14e2b8e0c5178078714f1 P 166bbea04fd64826a5aae830ef8bfbae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61c33320fe9844018afb635f9dee3641" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35639 } } peers { permanent_uuid: "166bbea04fd64826a5aae830ef8bfbae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43727 } } peers { permanent_uuid: "d73081e8b2f449138126fc26b5f3a5c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36989 } }
I20260926 01:10:04.920197 2712 leader_election.cc:288] T da53a21a08d14e2b8e0c5178078714f1 P 166bbea04fd64826a5aae830ef8bfbae [CANDIDATE]: Term 1 election: Requested vote from peers 61c33320fe9844018afb635f9dee3641 (127.0.98.1:35639), d73081e8b2f449138126fc26b5f3a5c2 (127.0.98.3:36989)
I20260926 01:10:04.920373 2274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da53a21a08d14e2b8e0c5178078714f1" candidate_uuid: "166bbea04fd64826a5aae830ef8bfbae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61c33320fe9844018afb635f9dee3641"
I20260926 01:10:04.920410 2554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da53a21a08d14e2b8e0c5178078714f1" candidate_uuid: "166bbea04fd64826a5aae830ef8bfbae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73081e8b2f449138126fc26b5f3a5c2"
I20260926 01:10:04.920472 2274 raft_consensus.cc:3037] T da53a21a08d14e2b8e0c5178078714f1 P 61c33320fe9844018afb635f9dee3641 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:10:04.920476 2554 raft_consensus.cc:3037] T da53a21a08d14e2b8e0c5178078714f1 P d73081e8b2f449138126fc26b5f3a5c2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:10:04.921237 2274 raft_consensus.cc:2454] T da53a21a08d14e2b8e0c5178078714f1 P 61c33320fe9844018afb635f9dee3641 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 166bbea04fd64826a5aae830ef8bfbae in term 1.
I20260926 01:10:04.921237 2554 raft_consensus.cc:2454] T da53a21a08d14e2b8e0c5178078714f1 P d73081e8b2f449138126fc26b5f3a5c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 166bbea04fd64826a5aae830ef8bfbae in term 1.
I20260926 01:10:04.921427 2348 leader_election.cc:302] T da53a21a08d14e2b8e0c5178078714f1 P 166bbea04fd64826a5aae830ef8bfbae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 166bbea04fd64826a5aae830ef8bfbae, 61c33320fe9844018afb635f9dee3641; no voters:
I20260926 01:10:04.921525 2712 raft_consensus.cc:2781] T da53a21a08d14e2b8e0c5178078714f1 P 166bbea04fd64826a5aae830ef8bfbae [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:10:04.921651 2712 raft_consensus.cc:686] T da53a21a08d14e2b8e0c5178078714f1 P 166bbea04fd64826a5aae830ef8bfbae [term 1 LEADER]: Becoming Leader. State: Replica: 166bbea04fd64826a5aae830ef8bfbae, State: Running, Role: LEADER
I20260926 01:10:04.921708 2712 consensus_queue.cc:237] T da53a21a08d14e2b8e0c5178078714f1 P 166bbea04fd64826a5aae830ef8bfbae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61c33320fe9844018afb635f9dee3641" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35639 } } peers { permanent_uuid: "166bbea04fd64826a5aae830ef8bfbae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43727 } } peers { permanent_uuid: "d73081e8b2f449138126fc26b5f3a5c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36989 } }
I20260926 01:10:04.922338 2116 catalog_manager.cc:5495] T da53a21a08d14e2b8e0c5178078714f1 P 166bbea04fd64826a5aae830ef8bfbae reported cstate change: term changed from 0 to 1, leader changed from <none> to 166bbea04fd64826a5aae830ef8bfbae (127.0.98.2). New cstate: current_term: 1 leader_uuid: "166bbea04fd64826a5aae830ef8bfbae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61c33320fe9844018afb635f9dee3641" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "166bbea04fd64826a5aae830ef8bfbae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43727 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d73081e8b2f449138126fc26b5f3a5c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36989 } health_report { overall_health: UNKNOWN } } }
I20260926 01:10:04.940184 2716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:04.938155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38984 (local address 127.0.98.2:43727)
0926 01:10:04.938197 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:04.938199 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:04.938206 (+ 7us) server_negotiation.cc:407] Connection header received
0926 01:10:04.938264 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:04.938265 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:04.938275 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:04.938313 (+ 38us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:10:04.938391 (+ 78us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:04.938855 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:04.939514 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:04.939762 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:04.939818 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:04.939983 (+ 165us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:10:04.940012 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:10:04.940049 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:04.940087 (+ 38us) server_negotiation.cc:299] Negotiation successful
0926 01:10:04.940108 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 01:10:04.942361 2554 raft_consensus.cc:1261] T da53a21a08d14e2b8e0c5178078714f1 P d73081e8b2f449138126fc26b5f3a5c2 [term 1 FOLLOWER]: Refusing update from remote peer 166bbea04fd64826a5aae830ef8bfbae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:10:04.942430 2274 raft_consensus.cc:1261] T da53a21a08d14e2b8e0c5178078714f1 P 61c33320fe9844018afb635f9dee3641 [term 1 FOLLOWER]: Refusing update from remote peer 166bbea04fd64826a5aae830ef8bfbae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:10:04.942631 2712 consensus_queue.cc:1035] T da53a21a08d14e2b8e0c5178078714f1 P 166bbea04fd64826a5aae830ef8bfbae [LEADER]: Connected to new peer: Peer: permanent_uuid: "d73081e8b2f449138126fc26b5f3a5c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:10:04.942734 2736 consensus_queue.cc:1035] T da53a21a08d14e2b8e0c5178078714f1 P 166bbea04fd64826a5aae830ef8bfbae [LEADER]: Connected to new peer: Peer: permanent_uuid: "61c33320fe9844018afb635f9dee3641" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 01:10:04.947196 2679 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260926 01:10:06.466813 2116 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49728:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:10:06.466941 2116 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49728:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:10:06.467648 2116 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 43da41c6d9f34b998e1c66111660121f: Sending DeleteTablet for 3 replicas of tablet da53a21a08d14e2b8e0c5178078714f1
I20260926 01:10:06.467861 2254 tablet_service.cc:1514] Processing DeleteTablet for tablet da53a21a08d14e2b8e0c5178078714f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:10:06 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53100
I20260926 01:10:06.467906 2394 tablet_service.cc:1514] Processing DeleteTablet for tablet da53a21a08d14e2b8e0c5178078714f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:10:06 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38970
I20260926 01:10:06.467931 2520 tablet_service.cc:1514] Processing DeleteTablet for tablet da53a21a08d14e2b8e0c5178078714f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:10:06 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42100
I20260926 01:10:06.468063 2765 tablet_replica.cc:330] T da53a21a08d14e2b8e0c5178078714f1 P 166bbea04fd64826a5aae830ef8bfbae: stopping tablet replica
I20260926 01:10:06.468094 2764 tablet_replica.cc:330] T da53a21a08d14e2b8e0c5178078714f1 P 61c33320fe9844018afb635f9dee3641: stopping tablet replica
I20260926 01:10:06.468111 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 2189
I20260926 01:10:06.468129 2765 raft_consensus.cc:2229] T da53a21a08d14e2b8e0c5178078714f1 P 166bbea04fd64826a5aae830ef8bfbae [term 1 LEADER]: Raft consensus shutting down.
I20260926 01:10:06.468146 2764 raft_consensus.cc:2229] T da53a21a08d14e2b8e0c5178078714f1 P 61c33320fe9844018afb635f9dee3641 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:10:06.469084 2765 raft_consensus.cc:2258] T da53a21a08d14e2b8e0c5178078714f1 P 166bbea04fd64826a5aae830ef8bfbae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:10:06.469322 2765 ts_tablet_manager.cc:1905] T da53a21a08d14e2b8e0c5178078714f1 P 166bbea04fd64826a5aae830ef8bfbae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:10:06.470973 2765 ts_tablet_manager.cc:1918] T da53a21a08d14e2b8e0c5178078714f1 P 166bbea04fd64826a5aae830ef8bfbae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 01:10:06.471030 2765 log.cc:1196] T da53a21a08d14e2b8e0c5178078714f1 P 166bbea04fd64826a5aae830ef8bfbae: Deleting WAL directory at /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384993574762-392-0/minicluster-data/ts-1/wal/wals/da53a21a08d14e2b8e0c5178078714f1
I20260926 01:10:06.471215 2765 ts_tablet_manager.cc:1939] T da53a21a08d14e2b8e0c5178078714f1 P 166bbea04fd64826a5aae830ef8bfbae: Deleting consensus metadata
I20260926 01:10:06.471503 2104 catalog_manager.cc:4841] TS 166bbea04fd64826a5aae830ef8bfbae (127.0.98.2:43727): tablet da53a21a08d14e2b8e0c5178078714f1 (table test-table [id=c90ecd42675a4eafb797d1debc38d230]) successfully deleted
W20260926 01:10:06.474080 2104 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35639 (error 108)
I20260926 01:10:06.474314 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 2329
I20260926 01:10:06.475044 2768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:06.474460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35639 (local address 127.0.0.1:53114)
0926 01:10:06.474948 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:06.475001 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":54,"threads_started":1}
W20260926 01:10:06.475253 2104 catalog_manager.cc:4584] TS 61c33320fe9844018afb635f9dee3641 (127.0.98.1:35639): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet da53a21a08d14e2b8e0c5178078714f1: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35639: connect: Connection refused (error 111)
I20260926 01:10:06.475391 2766 tablet_replica.cc:330] T da53a21a08d14e2b8e0c5178078714f1 P d73081e8b2f449138126fc26b5f3a5c2: stopping tablet replica
I20260926 01:10:06.475458 2766 raft_consensus.cc:2229] T da53a21a08d14e2b8e0c5178078714f1 P d73081e8b2f449138126fc26b5f3a5c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:10:06.475507 2766 raft_consensus.cc:2258] T da53a21a08d14e2b8e0c5178078714f1 P d73081e8b2f449138126fc26b5f3a5c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:10:06.475843 2766 ts_tablet_manager.cc:1905] T da53a21a08d14e2b8e0c5178078714f1 P d73081e8b2f449138126fc26b5f3a5c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:10:06.477425 2766 ts_tablet_manager.cc:1918] T da53a21a08d14e2b8e0c5178078714f1 P d73081e8b2f449138126fc26b5f3a5c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 01:10:06.477485 2766 log.cc:1196] T da53a21a08d14e2b8e0c5178078714f1 P d73081e8b2f449138126fc26b5f3a5c2: Deleting WAL directory at /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384993574762-392-0/minicluster-data/ts-2/wal/wals/da53a21a08d14e2b8e0c5178078714f1
I20260926 01:10:06.477675 2766 ts_tablet_manager.cc:1939] T da53a21a08d14e2b8e0c5178078714f1 P d73081e8b2f449138126fc26b5f3a5c2: Deleting consensus metadata
I20260926 01:10:06.478152 2103 catalog_manager.cc:4841] TS d73081e8b2f449138126fc26b5f3a5c2 (127.0.98.3:36989): tablet da53a21a08d14e2b8e0c5178078714f1 (table test-table [id=c90ecd42675a4eafb797d1debc38d230]) successfully deleted
I20260926 01:10:06.480275 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 2469
I20260926 01:10:06.485726 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 2085
2026-09-26T01:10:06Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5514 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384993574762-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 01:10:06 dist-test-slave-7b22 krb5kdc[2771](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 01:10:06 dist-test-slave-7b22 krb5kdc[2771](info): set up 2 sockets
Sep 26 01:10:06 dist-test-slave-7b22 krb5kdc[2771](info): commencing operation
krb5kdc: starting...
W20260926 01:10:07.433374 392 mini_kdc.cc:121] Time spent starting KDC: real 0.928s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384993574762-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384993574762-392-0/krb5kdc/test-user.keytab.
Sep 26 01:10:07 dist-test-slave-7b22 krb5kdc[2771](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385007, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T01:10:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:10:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384993574762-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384993574762-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 01:10:07.552980 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384993574762-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384993574762-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384993574762-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44869
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384993574762-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44869
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:10:07.635011 2784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:10:07.635250 2784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:10:07.635282 2784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:10:07.636488 2784 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:10:07.636533 2784 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:10:07.636552 2784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:10:07.636569 2784 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:10:07.636583 2784 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:10:07.638185 2784 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384993574762-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384993574762-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44869
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384993574762-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44869
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2784
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384993574762-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:10:07.638446 2784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:10:07.638653 2784 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 01:10:07.643488 2784 server_base.cc:1034] running on GCE node
W20260926 01:10:07.643488 2789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:10:07.643507 2790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:10:07.643546 2792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:10:07.644155 2784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:10:07.644469 2784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:10:07.645640 2784 hybrid_clock.cc:648] HybridClock initialized: now 1790385007645622 us; error 30 us; skew 500 ppm
Sep 26 01:10:07 dist-test-slave-7b22 krb5kdc[2771](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385007, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:10:07.648499 2784 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 01:10:07.649009 2784 webserver.cc:460] Webserver started at http://127.0.98.62:46677/ using document root <none> and password file <none>
I20260926 01:10:07.649245 2784 fs_manager.cc:359] Metadata directory not provided
I20260926 01:10:07.649299 2784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:10:07.649431 2784 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:10:07.650616 2784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384993574762-392-0/minicluster-data/master-0/data/instance:
uuid: "5af98a44686c4f4da7011eb1116ea177"
format_stamp: "Formatted at 2026-09-26 01:10:07 on dist-test-slave-7b22"
server_key: "ccaf07060bbc9c4227037e4048225f78"
server_key_iv: "8876226558602a29c82c8fb70de3e5e8"
server_key_version: "encryptionkey@0"
I20260926 01:10:07.650972 2784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384993574762-392-0/minicluster-data/master-0/wal/instance:
uuid: "5af98a44686c4f4da7011eb1116ea177"
format_stamp: "Formatted at 2026-09-26 01:10:07 on dist-test-slave-7b22"
server_key: "ccaf07060bbc9c4227037e4048225f78"
server_key_iv: "8876226558602a29c82c8fb70de3e5e8"
server_key_version: "encryptionkey@0"
I20260926 01:10:07.652370 2784 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 01:10:07.653198 2799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:07.653381 2784 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:10:07.653477 2784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384993574762-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384993574762-392-0/minicluster-data/master-0/wal
uuid: "5af98a44686c4f4da7011eb1116ea177"
format_stamp: "Formatted at 2026-09-26 01:10:07 on dist-test-slave-7b22"
server_key: "ccaf07060bbc9c4227037e4048225f78"
server_key_iv: "8876226558602a29c82c8fb70de3e5e8"
server_key_version: "encryptionkey@0"
I20260926 01:10:07.653538 2784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384993574762-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384993574762-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384993574762-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:10:07.672617 2784 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:10:07.674602 2784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:10:07.674736 2784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:10:07.678508 2784 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44869
I20260926 01:10:07.678552 2851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44869 every 8 connection(s)
I20260926 01:10:07.679035 2784 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
I20260926 01:10:07.679764 2852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:10:07.682063 2852 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5af98a44686c4f4da7011eb1116ea177: Bootstrap starting.
I20260926 01:10:07.682610 2852 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5af98a44686c4f4da7011eb1116ea177: Neither blocks nor log segments found. Creating new log.
I20260926 01:10:07.682832 2852 log.cc:824] T 00000000000000000000000000000000 P 5af98a44686c4f4da7011eb1116ea177: Log is configured to *not* fsync() on all Append() calls
I20260926 01:10:07.683423 2852 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5af98a44686c4f4da7011eb1116ea177: No bootstrap required, opened a new log
I20260926 01:10:07.684994 2852 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5af98a44686c4f4da7011eb1116ea177 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5af98a44686c4f4da7011eb1116ea177" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44869 } }
I20260926 01:10:07.685143 2852 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5af98a44686c4f4da7011eb1116ea177 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:10:07.685166 2852 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5af98a44686c4f4da7011eb1116ea177 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5af98a44686c4f4da7011eb1116ea177, State: Initialized, Role: FOLLOWER
I20260926 01:10:07.685285 2852 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5af98a44686c4f4da7011eb1116ea177 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5af98a44686c4f4da7011eb1116ea177" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44869 } }
I20260926 01:10:07.685361 2852 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5af98a44686c4f4da7011eb1116ea177 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:10:07.685397 2852 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5af98a44686c4f4da7011eb1116ea177 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:10:07.685431 2852 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5af98a44686c4f4da7011eb1116ea177 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:10:07.686291 2852 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5af98a44686c4f4da7011eb1116ea177 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5af98a44686c4f4da7011eb1116ea177" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44869 } }
I20260926 01:10:07.686378 2852 leader_election.cc:302] T 00000000000000000000000000000000 P 5af98a44686c4f4da7011eb1116ea177 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5af98a44686c4f4da7011eb1116ea177; no voters:
I20260926 01:10:07.686512 2852 leader_election.cc:288] T 00000000000000000000000000000000 P 5af98a44686c4f4da7011eb1116ea177 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:10:07.686558 2855 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5af98a44686c4f4da7011eb1116ea177 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:10:07.686683 2855 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5af98a44686c4f4da7011eb1116ea177 [term 1 LEADER]: Becoming Leader. State: Replica: 5af98a44686c4f4da7011eb1116ea177, State: Running, Role: LEADER
I20260926 01:10:07.686772 2852 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5af98a44686c4f4da7011eb1116ea177 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:10:07.686834 2855 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5af98a44686c4f4da7011eb1116ea177 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5af98a44686c4f4da7011eb1116ea177" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44869 } }
I20260926 01:10:07.687167 2857 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5af98a44686c4f4da7011eb1116ea177 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5af98a44686c4f4da7011eb1116ea177. Latest consensus state: current_term: 1 leader_uuid: "5af98a44686c4f4da7011eb1116ea177" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5af98a44686c4f4da7011eb1116ea177" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44869 } } }
I20260926 01:10:07.687161 2856 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5af98a44686c4f4da7011eb1116ea177 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5af98a44686c4f4da7011eb1116ea177" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5af98a44686c4f4da7011eb1116ea177" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44869 } } }
I20260926 01:10:07.687254 2857 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5af98a44686c4f4da7011eb1116ea177 [sys.catalog]: This master's current role is: LEADER
I20260926 01:10:07.687261 2856 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5af98a44686c4f4da7011eb1116ea177 [sys.catalog]: This master's current role is: LEADER
I20260926 01:10:07.687443 2863 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:10:07.687992 2863 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:10:07.688011 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 2784
I20260926 01:10:07.688086 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384993574762-392-0/minicluster-data/master-0/wal/instance
I20260926 01:10:07.688265 392 external_mini_cluster.cc:1428] Setting key e6852d2c2196b6680d29546a62087552
I20260926 01:10:07.689759 2863 catalog_manager.cc:1347] Generated new cluster ID: 73737feae9ee4dd2a563fb006a9a828c
I20260926 01:10:07.689805 2863 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 01:10:07 dist-test-slave-7b22 krb5kdc[2771](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385007, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:10:07.699589 2874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:07.688928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35580 (local address 127.0.98.62:44869)
0926 01:10:07.689060 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:07.689069 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:07.689097 (+ 28us) server_negotiation.cc:407] Connection header received
0926 01:10:07.689229 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:07.689235 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:07.689302 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:07.689443 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:07.690175 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:07.691543 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:07.692574 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:07.693183 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:07.693553 (+ 370us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:07.695882 (+ 2329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:10:07.695886 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:10:07.695886 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:10:07.695899 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:10:07.697705 (+ 1806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:07.698109 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:07.698112 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:07.698112 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:07.698240 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:07.698410 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:07.698411 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:07.698411 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:07.698698 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:10:07.699144 (+ 446us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:07.699212 (+ 68us) server_negotiation.cc:299] Negotiation successful
0926 01:10:07.699350 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":68,"threads_started":1}
I20260926 01:10:07.704028 2863 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:10:07.704527 2863 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:10:07.710942 2863 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5af98a44686c4f4da7011eb1116ea177: Generated new TSK 0
I20260926 01:10:07.711149 2863 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384993574762-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384993574762-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 01:10:07.741833 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384993574762-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384993574762-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384993574762-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384993574762-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44869
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384993574762-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:10:07.856258 2878 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:10:07.856477 2878 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:10:07.856500 2878 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:10:07.857713 2878 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:10:07.857758 2878 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:10:07.857785 2878 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 01:10:07.859349 2878 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384993574762-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384993574762-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384993574762-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384993574762-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44869
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2878
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384993574762-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:10:07.859548 2878 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:10:07.859727 2878 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:10:07.864640 2883 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:10:07.864648 2886 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:10:07.864665 2884 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:10:07.864822 2878 server_base.cc:1034] running on GCE node
I20260926 01:10:07.865147 2878 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:10:07.865386 2878 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:10:07.866515 2878 hybrid_clock.cc:648] HybridClock initialized: now 1790385007866503 us; error 29 us; skew 500 ppm
Sep 26 01:10:07 dist-test-slave-7b22 krb5kdc[2771](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385007, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:10:07.868894 2878 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 01:10:07.869302 2878 webserver.cc:460] Webserver started at http://127.0.98.1:46531/ using document root <none> and password file <none>
I20260926 01:10:07.869477 2878 fs_manager.cc:359] Metadata directory not provided
I20260926 01:10:07.869515 2878 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:10:07.869611 2878 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:10:07.870676 2878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384993574762-392-0/minicluster-data/ts-0/data/instance:
uuid: "06519f029e084a568bd1f27616c72625"
format_stamp: "Formatted at 2026-09-26 01:10:07 on dist-test-slave-7b22"
server_key: "0122cb4d39b9d22440798c25da26ae3d"
server_key_iv: "ce9d9448b5b3d2d8009af71fac724625"
server_key_version: "encryptionkey@0"
I20260926 01:10:07.871004 2878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384993574762-392-0/minicluster-data/ts-0/wal/instance:
uuid: "06519f029e084a568bd1f27616c72625"
format_stamp: "Formatted at 2026-09-26 01:10:07 on dist-test-slave-7b22"
server_key: "0122cb4d39b9d22440798c25da26ae3d"
server_key_iv: "ce9d9448b5b3d2d8009af71fac724625"
server_key_version: "encryptionkey@0"
I20260926 01:10:07.872315 2878 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 01:10:07.872975 2893 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:07.873138 2878 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 01:10:07.873224 2878 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384993574762-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384993574762-392-0/minicluster-data/ts-0/wal
uuid: "06519f029e084a568bd1f27616c72625"
format_stamp: "Formatted at 2026-09-26 01:10:07 on dist-test-slave-7b22"
server_key: "0122cb4d39b9d22440798c25da26ae3d"
server_key_iv: "ce9d9448b5b3d2d8009af71fac724625"
server_key_version: "encryptionkey@0"
I20260926 01:10:07.873294 2878 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384993574762-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384993574762-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384993574762-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:10:07.885033 2878 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:10:07.887570 2878 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:10:07.887710 2878 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:10:07.887964 2878 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:10:07.888304 2878 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:10:07.888370 2878 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:07.888403 2878 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:10:07.888427 2878 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:07.894057 2878 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34441
I20260926 01:10:07.894114 3006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34441 every 8 connection(s)
I20260926 01:10:07.894445 2878 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384993574762-392-0/minicluster-data/ts-0/data/info.pb
I20260926 01:10:07.897612 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 2878
I20260926 01:10:07.897706 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384993574762-392-0/minicluster-data/ts-0/wal/instance
I20260926 01:10:07.897902 392 external_mini_cluster.cc:1428] Setting key 2b08e1671393f80e6a53a60ff00c8417
Sep 26 01:10:07 dist-test-slave-7b22 krb5kdc[2771](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385007, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:10:07.904408 2874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:07.895207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37873 (local address 127.0.98.62:44869)
0926 01:10:07.895278 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:07.895280 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:07.895465 (+ 185us) server_negotiation.cc:407] Connection header received
0926 01:10:07.896476 (+ 1011us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:07.896478 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:07.896495 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:07.896546 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:07.897191 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:07.897701 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:07.898906 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:07.899193 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:07.899269 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:07.901491 (+ 2222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:10:07.901492 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:10:07.901493 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:10:07.901499 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:10:07.902897 (+ 1398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:07.903391 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:07.903391 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:07.903392 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:07.903441 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:07.903714 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:07.903714 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:07.903714 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:07.903835 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:10:07.903882 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:07.904247 (+ 365us) server_negotiation.cc:299] Negotiation successful
0926 01:10:07.904336 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 01:10:07.904646 3009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:07.895186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44869 (local address 127.0.98.1:37873)
0926 01:10:07.895348 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:07.895388 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:07.896356 (+ 968us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:07.896622 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:07.896625 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:07.896896 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:10:07.897094 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:07.897099 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:07.897811 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:07.897812 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:07.898824 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:07.898827 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:07.899297 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:07.899298 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:07.899420 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:07.899957 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:10:07.899966 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:10:07.901383 (+ 1417us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:10:07.902982 (+ 1599us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:10:07.902990 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:10:07.902993 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:10:07.903300 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:10:07.903496 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:10:07.903497 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:10:07.903498 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:10:07.903656 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:10:07.903892 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:10:07.903898 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:10:07.904080 (+ 182us) client_negotiation.cc:769] Sending connection context
0926 01:10:07.904203 (+ 123us) client_negotiation.cc:240] Negotiation successful
0926 01:10:07.904383 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":71,"threads_started":1}
I20260926 01:10:07.905341 3007 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44869
I20260926 01:10:07.905444 3007 heartbeater.cc:464] Registering TS with master...
I20260926 01:10:07.905625 3007 heartbeater.cc:511] Master 127.0.98.62:44869 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260926 01:10:07.906180 2816 ts_manager.cc:194] Registered new tserver with Master: 06519f029e084a568bd1f27616c72625 (127.0.98.1:34441)
I20260926 01:10:07.906808 2816 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:37873
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384993574762-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384993574762-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 01:10:07.925499 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384993574762-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384993574762-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384993574762-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384993574762-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44869
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384993574762-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:10:08.006444 3012 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:10:08.006721 3012 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:10:08.006758 3012 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:10:08.008081 3012 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:10:08.008136 3012 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:10:08.008173 3012 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 01:10:08.009893 3012 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384993574762-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384993574762-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384993574762-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384993574762-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44869
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3012
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384993574762-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:10:08.010156 3012 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:10:08.010377 3012 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:10:08.015233 3017 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:10:08.015331 3020 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:10:08.015333 3018 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:10:08.015776 3012 server_base.cc:1034] running on GCE node
I20260926 01:10:08.016001 3012 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:10:08.016216 3012 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:10:08.017369 3012 hybrid_clock.cc:648] HybridClock initialized: now 1790385008017350 us; error 31 us; skew 500 ppm
Sep 26 01:10:08 dist-test-slave-7b22 krb5kdc[2771](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385008, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:10:08.019937 3012 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 01:10:08.020438 3012 webserver.cc:460] Webserver started at http://127.0.98.2:33077/ using document root <none> and password file <none>
I20260926 01:10:08.020661 3012 fs_manager.cc:359] Metadata directory not provided
I20260926 01:10:08.020705 3012 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:10:08.020816 3012 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:10:08.021920 3012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384993574762-392-0/minicluster-data/ts-1/data/instance:
uuid: "6f5511709fef4db081a7e3bcaa1af466"
format_stamp: "Formatted at 2026-09-26 01:10:08 on dist-test-slave-7b22"
server_key: "959a8d76c09077c05fb73993f721a998"
server_key_iv: "5e2989c9560c6d1437b890d1ca7e1551"
server_key_version: "encryptionkey@0"
I20260926 01:10:08.022257 3012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384993574762-392-0/minicluster-data/ts-1/wal/instance:
uuid: "6f5511709fef4db081a7e3bcaa1af466"
format_stamp: "Formatted at 2026-09-26 01:10:08 on dist-test-slave-7b22"
server_key: "959a8d76c09077c05fb73993f721a998"
server_key_iv: "5e2989c9560c6d1437b890d1ca7e1551"
server_key_version: "encryptionkey@0"
I20260926 01:10:08.023561 3012 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 01:10:08.024284 3027 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:08.024466 3012 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:08.024554 3012 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384993574762-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384993574762-392-0/minicluster-data/ts-1/wal
uuid: "6f5511709fef4db081a7e3bcaa1af466"
format_stamp: "Formatted at 2026-09-26 01:10:08 on dist-test-slave-7b22"
server_key: "959a8d76c09077c05fb73993f721a998"
server_key_iv: "5e2989c9560c6d1437b890d1ca7e1551"
server_key_version: "encryptionkey@0"
I20260926 01:10:08.024631 3012 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384993574762-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384993574762-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384993574762-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:10:08.038331 3012 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:10:08.040598 3012 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:10:08.040725 3012 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:10:08.040961 3012 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:10:08.041276 3012 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:10:08.041334 3012 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:08.041356 3012 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:10:08.041379 3012 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:08.046777 3012 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36517
I20260926 01:10:08.046856 3140 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36517 every 8 connection(s)
I20260926 01:10:08.047120 3012 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384993574762-392-0/minicluster-data/ts-1/data/info.pb
I20260926 01:10:08.050231 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 3012
I20260926 01:10:08.050318 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384993574762-392-0/minicluster-data/ts-1/wal/instance
I20260926 01:10:08.050477 392 external_mini_cluster.cc:1428] Setting key bfb0a75ceaba5dea759d13b9dd0b83b2
Sep 26 01:10:08 dist-test-slave-7b22 krb5kdc[2771](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385008, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:10:08.057238 2874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:08.047869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45987 (local address 127.0.98.62:44869)
0926 01:10:08.047930 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:08.047933 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:08.048101 (+ 168us) server_negotiation.cc:407] Connection header received
0926 01:10:08.049165 (+ 1064us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:08.049166 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:08.049178 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:08.049243 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:08.050015 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:08.050572 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:08.051914 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:08.052104 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:08.052157 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:08.054567 (+ 2410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:10:08.054568 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:10:08.054568 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:10:08.054575 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:10:08.055944 (+ 1369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:08.056315 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:08.056316 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:08.056316 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:08.056371 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:08.056589 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:08.056590 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:08.056590 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:08.056701 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:10:08.056756 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:08.057056 (+ 300us) server_negotiation.cc:299] Negotiation successful
0926 01:10:08.057143 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 01:10:08.057430 3143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:08.047842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44869 (local address 127.0.98.2:45987)
0926 01:10:08.047986 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:08.048022 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:08.049041 (+ 1019us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:08.049347 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:08.049350 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:08.049691 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:10:08.049945 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:08.049951 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:08.050657 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:08.050657 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:08.051834 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:08.051836 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:08.052174 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:08.052175 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:08.052312 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:08.052854 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:10:08.052863 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:10:08.054464 (+ 1601us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:10:08.056022 (+ 1558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:10:08.056029 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:10:08.056031 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:10:08.056249 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:10:08.056424 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:10:08.056425 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:10:08.056425 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:10:08.056535 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:10:08.056754 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:10:08.056760 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:10:08.056940 (+ 180us) client_negotiation.cc:769] Sending connection context
0926 01:10:08.057033 (+ 93us) client_negotiation.cc:240] Negotiation successful
0926 01:10:08.057195 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":72,"threads_started":1}
I20260926 01:10:08.057885 3141 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44869
I20260926 01:10:08.057978 3141 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260926 01:10:08.058161 3141 heartbeater.cc:511] Master 127.0.98.62:44869 requested a full tablet report, sending...
I20260926 01:10:08.058548 2816 ts_manager.cc:194] Registered new tserver with Master: 6f5511709fef4db081a7e3bcaa1af466 (127.0.98.2:36517)
I20260926 01:10:08.058911 2816 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:45987
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384993574762-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384993574762-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 01:10:08.076985 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384993574762-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384993574762-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384993574762-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384993574762-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44869
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384993574762-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:10:08.158177 3146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:10:08.158396 3146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:10:08.158420 3146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:10:08.159689 3146 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:10:08.159739 3146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:10:08.159766 3146 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 01:10:08.161275 3146 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384993574762-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384993574762-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384993574762-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384993574762-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44869
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3146
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384993574762-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:10:08.161463 3146 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:10:08.161659 3146 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:10:08.166761 3154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:10:08.166793 3152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:10:08.166805 3146 server_base.cc:1034] running on GCE node
W20260926 01:10:08.166769 3151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:10:08.167146 3146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:10:08.167425 3146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:10:08.168641 3146 hybrid_clock.cc:648] HybridClock initialized: now 1790385008168616 us; error 40 us; skew 500 ppm
Sep 26 01:10:08 dist-test-slave-7b22 krb5kdc[2771](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385008, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:10:08.171540 3146 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 01:10:08.172132 3146 webserver.cc:460] Webserver started at http://127.0.98.3:34333/ using document root <none> and password file <none>
I20260926 01:10:08.172346 3146 fs_manager.cc:359] Metadata directory not provided
I20260926 01:10:08.172394 3146 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:10:08.172519 3146 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:10:08.173594 3146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384993574762-392-0/minicluster-data/ts-2/data/instance:
uuid: "465962fbd0694301843ef95367844b9d"
format_stamp: "Formatted at 2026-09-26 01:10:08 on dist-test-slave-7b22"
server_key: "bdb2bdf846687714b3a5bec6b065fc8b"
server_key_iv: "2210ae13f42af2aa67f4d5f56ec4cad8"
server_key_version: "encryptionkey@0"
I20260926 01:10:08.173961 3146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384993574762-392-0/minicluster-data/ts-2/wal/instance:
uuid: "465962fbd0694301843ef95367844b9d"
format_stamp: "Formatted at 2026-09-26 01:10:08 on dist-test-slave-7b22"
server_key: "bdb2bdf846687714b3a5bec6b065fc8b"
server_key_iv: "2210ae13f42af2aa67f4d5f56ec4cad8"
server_key_version: "encryptionkey@0"
I20260926 01:10:08.175336 3146 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 01:10:08.176095 3161 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:08.176265 3146 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 01:10:08.176352 3146 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384993574762-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384993574762-392-0/minicluster-data/ts-2/wal
uuid: "465962fbd0694301843ef95367844b9d"
format_stamp: "Formatted at 2026-09-26 01:10:08 on dist-test-slave-7b22"
server_key: "bdb2bdf846687714b3a5bec6b065fc8b"
server_key_iv: "2210ae13f42af2aa67f4d5f56ec4cad8"
server_key_version: "encryptionkey@0"
I20260926 01:10:08.176416 3146 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384993574762-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384993574762-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384993574762-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:10:08.186789 3146 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:10:08.189043 3146 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:10:08.189165 3146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:10:08.189406 3146 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:10:08.189725 3146 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:10:08.189759 3146 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:08.189790 3146 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:10:08.189812 3146 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:08.195297 3146 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37183
I20260926 01:10:08.195346 3274 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37183 every 8 connection(s)
I20260926 01:10:08.195657 3146 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384993574762-392-0/minicluster-data/ts-2/data/info.pb
Sep 26 01:10:08 dist-test-slave-7b22 krb5kdc[2771](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385008, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:10:08.201719 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 3146
I20260926 01:10:08.201807 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384993574762-392-0/minicluster-data/ts-2/wal/instance
I20260926 01:10:08.201983 392 external_mini_cluster.cc:1428] Setting key 979897d26c425d3e998f94ec9a4fd6a1
I20260926 01:10:08.204754 2874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:08.196360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49915 (local address 127.0.98.62:44869)
0926 01:10:08.196419 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:08.196420 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:08.196612 (+ 192us) server_negotiation.cc:407] Connection header received
0926 01:10:08.197522 (+ 910us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:08.197523 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:08.197532 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:08.197596 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:08.198299 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:08.198814 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:08.199788 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:08.199992 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:08.200055 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:08.202271 (+ 2216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:10:08.202273 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:10:08.202274 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:10:08.202282 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:10:08.203521 (+ 1239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:08.203880 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:08.203881 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:08.203882 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:08.203934 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:08.204156 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:08.204157 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:08.204157 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:08.204265 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:10:08.204318 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:08.204584 (+ 266us) server_negotiation.cc:299] Negotiation successful
0926 01:10:08.204664 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 01:10:08.204927 3277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:08.196324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44869 (local address 127.0.98.3:49915)
0926 01:10:08.196490 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:08.196533 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:08.197435 (+ 902us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:08.197669 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:08.197673 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:08.197985 (+ 312us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:10:08.198183 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:08.198188 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:08.198884 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:08.198885 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:08.199699 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:08.199701 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:08.200062 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:08.200063 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:08.200164 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:08.200740 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:10:08.200749 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:10:08.202195 (+ 1446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:10:08.203593 (+ 1398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:10:08.203601 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:10:08.203603 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:10:08.203817 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:10:08.203986 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:10:08.203987 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:10:08.203987 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:10:08.204102 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:10:08.204321 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:10:08.204327 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:10:08.204462 (+ 135us) client_negotiation.cc:769] Sending connection context
0926 01:10:08.204551 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 01:10:08.204705 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":84,"threads_started":1}
I20260926 01:10:08.205391 3275 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44869
I20260926 01:10:08.205487 3275 heartbeater.cc:464] Registering TS with master...
I20260926 01:10:08.205634 3275 heartbeater.cc:511] Master 127.0.98.62:44869 requested a full tablet report, sending...
I20260926 01:10:08.205994 2816 ts_manager.cc:194] Registered new tserver with Master: 465962fbd0694301843ef95367844b9d (127.0.98.3:37183)
I20260926 01:10:08.206351 2816 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:49915
I20260926 01:10:08.213856 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 01:10:08.225076 2874 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:10:08.224136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35586 (local address 127.0.98.62:44869)
0926 01:10:08.224223 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:08.224226 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:08.224283 (+ 57us) server_negotiation.cc:407] Connection header received
0926 01:10:08.224356 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:08.224357 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:08.224368 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:08.224418 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:08.225018 (+ 600us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35586: BlockingRecv error: recv got EOF from 127.0.0.1:35586 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 01:10:08.226018 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 2878
I20260926 01:10:08.230444 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 3012
I20260926 01:10:08.234493 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 3146
I20260926 01:10:08.238811 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 2784
2026-09-26T01:10:08Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1751 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790384993574762-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 01:10:08 dist-test-slave-7b22 krb5kdc[3287](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 01:10:08 dist-test-slave-7b22 krb5kdc[3287](info): set up 2 sockets
Sep 26 01:10:08 dist-test-slave-7b22 krb5kdc[3287](info): commencing operation
krb5kdc: starting...
W20260926 01:10:09.185696 392 mini_kdc.cc:121] Time spent starting KDC: real 0.928s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790384993574762-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790384993574762-392-0/krb5kdc/test-user.keytab.
Sep 26 01:10:09 dist-test-slave-7b22 krb5kdc[3287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385009, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T01:10:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:10:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790384993574762-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790384993574762-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 01:10:09.305130 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790384993574762-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790384993574762-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790384993574762-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32771
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790384993574762-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:32771
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:10:09.386718 3300 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:10:09.387066 3300 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:10:09.387092 3300 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:10:09.388486 3300 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:10:09.388549 3300 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:10:09.388563 3300 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:10:09.388576 3300 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:10:09.388586 3300 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:10:09.390368 3300 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790384993574762-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790384993574762-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:32771
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790384993574762-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32771
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3300
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790384993574762-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:10:09.390636 3300 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:10:09.390888 3300 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:10:09.396063 3308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:10:09.396358 3305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:10:09.396356 3300 server_base.cc:1034] running on GCE node
W20260926 01:10:09.396359 3306 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:10:09.396929 3300 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:10:09.397248 3300 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:10:09.398439 3300 hybrid_clock.cc:648] HybridClock initialized: now 1790385009398417 us; error 31 us; skew 500 ppm
Sep 26 01:10:09 dist-test-slave-7b22 krb5kdc[3287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385009, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:10:09.401340 3300 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 01:10:09.402187 3300 webserver.cc:460] Webserver started at http://127.0.98.62:40123/ using document root <none> and password file <none>
I20260926 01:10:09.402444 3300 fs_manager.cc:359] Metadata directory not provided
I20260926 01:10:09.402499 3300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:10:09.402652 3300 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:10:09.403815 3300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790384993574762-392-0/minicluster-data/master-0/data/instance:
uuid: "e459a04b23ec4331bffa1a8f6ab2d032"
format_stamp: "Formatted at 2026-09-26 01:10:09 on dist-test-slave-7b22"
server_key: "5d7246e6273b028faa3214a6ed758e72"
server_key_iv: "e0c5a881787606b08c2b500370fa1cd2"
server_key_version: "encryptionkey@0"
I20260926 01:10:09.404196 3300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790384993574762-392-0/minicluster-data/master-0/wal/instance:
uuid: "e459a04b23ec4331bffa1a8f6ab2d032"
format_stamp: "Formatted at 2026-09-26 01:10:09 on dist-test-slave-7b22"
server_key: "5d7246e6273b028faa3214a6ed758e72"
server_key_iv: "e0c5a881787606b08c2b500370fa1cd2"
server_key_version: "encryptionkey@0"
I20260926 01:10:09.405596 3300 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 01:10:09.406563 3315 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:09.406880 3300 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:10:09.406997 3300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790384993574762-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790384993574762-392-0/minicluster-data/master-0/wal
uuid: "e459a04b23ec4331bffa1a8f6ab2d032"
format_stamp: "Formatted at 2026-09-26 01:10:09 on dist-test-slave-7b22"
server_key: "5d7246e6273b028faa3214a6ed758e72"
server_key_iv: "e0c5a881787606b08c2b500370fa1cd2"
server_key_version: "encryptionkey@0"
I20260926 01:10:09.407076 3300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790384993574762-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790384993574762-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790384993574762-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:10:09.432086 3300 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:10:09.434480 3300 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:10:09.434612 3300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:10:09.438512 3300 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:32771
I20260926 01:10:09.438555 3367 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:32771 every 8 connection(s)
I20260926 01:10:09.438916 3300 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
I20260926 01:10:09.439813 3368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:10:09.440413 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 3300
I20260926 01:10:09.440505 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790384993574762-392-0/minicluster-data/master-0/wal/instance
I20260926 01:10:09.440661 392 external_mini_cluster.cc:1428] Setting key 77586ccc0d1128a580183e8cc75fa458
I20260926 01:10:09.442668 3368 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e459a04b23ec4331bffa1a8f6ab2d032: Bootstrap starting.
I20260926 01:10:09.443331 3368 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e459a04b23ec4331bffa1a8f6ab2d032: Neither blocks nor log segments found. Creating new log.
I20260926 01:10:09.443604 3368 log.cc:824] T 00000000000000000000000000000000 P e459a04b23ec4331bffa1a8f6ab2d032: Log is configured to *not* fsync() on all Append() calls
I20260926 01:10:09.444388 3368 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e459a04b23ec4331bffa1a8f6ab2d032: No bootstrap required, opened a new log
I20260926 01:10:09.446523 3368 raft_consensus.cc:348] T 00000000000000000000000000000000 P e459a04b23ec4331bffa1a8f6ab2d032 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e459a04b23ec4331bffa1a8f6ab2d032" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32771 } }
I20260926 01:10:09.446753 3368 raft_consensus.cc:374] T 00000000000000000000000000000000 P e459a04b23ec4331bffa1a8f6ab2d032 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:10:09.446791 3368 raft_consensus.cc:729] T 00000000000000000000000000000000 P e459a04b23ec4331bffa1a8f6ab2d032 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e459a04b23ec4331bffa1a8f6ab2d032, State: Initialized, Role: FOLLOWER
I20260926 01:10:09.446931 3368 consensus_queue.cc:260] T 00000000000000000000000000000000 P e459a04b23ec4331bffa1a8f6ab2d032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e459a04b23ec4331bffa1a8f6ab2d032" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32771 } }
I20260926 01:10:09.447008 3368 raft_consensus.cc:388] T 00000000000000000000000000000000 P e459a04b23ec4331bffa1a8f6ab2d032 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:10:09.447041 3368 raft_consensus.cc:482] T 00000000000000000000000000000000 P e459a04b23ec4331bffa1a8f6ab2d032 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:10:09.447103 3368 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e459a04b23ec4331bffa1a8f6ab2d032 [term 0 FOLLOWER]: Advancing to term 1
Sep 26 01:10:09 dist-test-slave-7b22 krb5kdc[3287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385009, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:10:09.447757 3368 raft_consensus.cc:504] T 00000000000000000000000000000000 P e459a04b23ec4331bffa1a8f6ab2d032 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e459a04b23ec4331bffa1a8f6ab2d032" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32771 } }
I20260926 01:10:09.447862 3368 leader_election.cc:302] T 00000000000000000000000000000000 P e459a04b23ec4331bffa1a8f6ab2d032 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e459a04b23ec4331bffa1a8f6ab2d032; no voters:
I20260926 01:10:09.448036 3368 leader_election.cc:288] T 00000000000000000000000000000000 P e459a04b23ec4331bffa1a8f6ab2d032 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:10:09.448094 3373 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e459a04b23ec4331bffa1a8f6ab2d032 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:10:09.448347 3368 sys_catalog.cc:564] T 00000000000000000000000000000000 P e459a04b23ec4331bffa1a8f6ab2d032 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:10:09.448603 3373 raft_consensus.cc:686] T 00000000000000000000000000000000 P e459a04b23ec4331bffa1a8f6ab2d032 [term 1 LEADER]: Becoming Leader. State: Replica: e459a04b23ec4331bffa1a8f6ab2d032, State: Running, Role: LEADER
I20260926 01:10:09.448706 3373 consensus_queue.cc:237] T 00000000000000000000000000000000 P e459a04b23ec4331bffa1a8f6ab2d032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e459a04b23ec4331bffa1a8f6ab2d032" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32771 } }
I20260926 01:10:09.449349 3373 sys_catalog.cc:455] T 00000000000000000000000000000000 P e459a04b23ec4331bffa1a8f6ab2d032 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e459a04b23ec4331bffa1a8f6ab2d032. Latest consensus state: current_term: 1 leader_uuid: "e459a04b23ec4331bffa1a8f6ab2d032" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e459a04b23ec4331bffa1a8f6ab2d032" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32771 } } }
I20260926 01:10:09.449437 3373 sys_catalog.cc:458] T 00000000000000000000000000000000 P e459a04b23ec4331bffa1a8f6ab2d032 [sys.catalog]: This master's current role is: LEADER
W20260926 01:10:09.450197 3388 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e459a04b23ec4331bffa1a8f6ab2d032: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 01:10:09.450253 3388 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 01:10:09.450627 3375 sys_catalog.cc:455] T 00000000000000000000000000000000 P e459a04b23ec4331bffa1a8f6ab2d032 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e459a04b23ec4331bffa1a8f6ab2d032" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e459a04b23ec4331bffa1a8f6ab2d032" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32771 } } }
I20260926 01:10:09.450700 3375 sys_catalog.cc:458] T 00000000000000000000000000000000 P e459a04b23ec4331bffa1a8f6ab2d032 [sys.catalog]: This master's current role is: LEADER
I20260926 01:10:09.450920 3386 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:10:09.451102 3386 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:10:09.452143 3371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:09.441194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37006 (local address 127.0.98.62:32771)
0926 01:10:09.441389 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:09.441398 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:09.441425 (+ 27us) server_negotiation.cc:407] Connection header received
0926 01:10:09.441890 (+ 465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:09.441900 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:09.441962 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:09.442097 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:09.442787 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:09.443872 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:09.444668 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:09.445097 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:09.445197 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:09.447885 (+ 2688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:10:09.447890 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:10:09.447891 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:10:09.447906 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:10:09.450338 (+ 2432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:09.450842 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:09.450845 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:09.450846 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:09.450958 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:09.451170 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:09.451171 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:09.451171 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:09.451478 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:10:09.451642 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:09.451776 (+ 134us) server_negotiation.cc:299] Negotiation successful
0926 01:10:09.451923 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":85,"threads_started":1}
I20260926 01:10:09.452739 3386 catalog_manager.cc:1347] Generated new cluster ID: 51df2866aa2d4822be75c3e41aef313e
I20260926 01:10:09.452828 3386 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 01:10:09.453074 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 01:10:09.475348 3371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:09.471034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37022 (local address 127.0.98.62:32771)
0926 01:10:09.471079 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:09.471081 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:09.471228 (+ 147us) server_negotiation.cc:407] Connection header received
0926 01:10:09.471269 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:09.471270 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:09.471283 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:09.471349 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:09.471774 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:09.472565 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:09.473594 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:09.474000 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:09.474087 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:09.474819 (+ 732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:10:09.474821 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:10:09.474821 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:10:09.475080 (+ 259us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:10:09.475102 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:10:09.475109 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:10:09.475170 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:09.475226 (+ 56us) server_negotiation.cc:299] Negotiation successful
0926 01:10:09.475256 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 01:10:09.483698 3386 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:10:09.484548 3386 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:10:09.488286 3386 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e459a04b23ec4331bffa1a8f6ab2d032: Generated new TSK 0
I20260926 01:10:09.488507 3386 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 01:10:09.494170 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 3300
2026-09-26T01:10:09Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1254 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384993574762-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 01:10:09 dist-test-slave-7b22 krb5kdc[3401](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 01:10:09 dist-test-slave-7b22 krb5kdc[3401](info): set up 2 sockets
Sep 26 01:10:09 dist-test-slave-7b22 krb5kdc[3401](info): commencing operation
krb5kdc: starting...
W20260926 01:10:10.445561 392 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384993574762-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384993574762-392-0/krb5kdc/test-user.keytab.
Sep 26 01:10:10 dist-test-slave-7b22 krb5kdc[3401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385010, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T01:10:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:10:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384993574762-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384993574762-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 01:10:10.566851 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384993574762-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384993574762-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384993574762-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38895
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384993574762-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38895
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:10:10.648497 3414 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:10:10.648744 3414 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:10:10.648777 3414 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:10:10.650095 3414 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:10:10.650138 3414 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:10:10.650156 3414 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:10:10.650173 3414 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:10:10.650187 3414 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:10:10.651690 3414 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384993574762-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384993574762-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38895
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384993574762-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38895
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3414
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384993574762-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:10:10.651939 3414 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:10:10.652151 3414 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:10:10.656805 3419 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:10:10.656805 3422 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:10:10.656819 3420 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:10:10.657265 3414 server_base.cc:1034] running on GCE node
I20260926 01:10:10.657490 3414 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:10:10.657778 3414 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:10:10.658949 3414 hybrid_clock.cc:648] HybridClock initialized: now 1790385010658931 us; error 30 us; skew 500 ppm
Sep 26 01:10:10 dist-test-slave-7b22 krb5kdc[3401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385010, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:10:10.664773 3414 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 01:10:10.665205 3414 webserver.cc:460] Webserver started at http://127.0.98.62:40283/ using document root <none> and password file <none>
I20260926 01:10:10.665405 3414 fs_manager.cc:359] Metadata directory not provided
I20260926 01:10:10.665447 3414 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:10:10.665537 3414 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:10:10.666620 3414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384993574762-392-0/minicluster-data/master-0/data/instance:
uuid: "9f86b7c9fbe5429eb7a4b2e26b576281"
format_stamp: "Formatted at 2026-09-26 01:10:10 on dist-test-slave-7b22"
server_key: "39f7226ee8c9d6ac9425660a51f8311c"
server_key_iv: "0257b35ed5a88c259fadcf04289d596b"
server_key_version: "encryptionkey@0"
I20260926 01:10:10.666931 3414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384993574762-392-0/minicluster-data/master-0/wal/instance:
uuid: "9f86b7c9fbe5429eb7a4b2e26b576281"
format_stamp: "Formatted at 2026-09-26 01:10:10 on dist-test-slave-7b22"
server_key: "39f7226ee8c9d6ac9425660a51f8311c"
server_key_iv: "0257b35ed5a88c259fadcf04289d596b"
server_key_version: "encryptionkey@0"
I20260926 01:10:10.668216 3414 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 01:10:10.668977 3429 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:10.669165 3414 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 01:10:10.669253 3414 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384993574762-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384993574762-392-0/minicluster-data/master-0/wal
uuid: "9f86b7c9fbe5429eb7a4b2e26b576281"
format_stamp: "Formatted at 2026-09-26 01:10:10 on dist-test-slave-7b22"
server_key: "39f7226ee8c9d6ac9425660a51f8311c"
server_key_iv: "0257b35ed5a88c259fadcf04289d596b"
server_key_version: "encryptionkey@0"
I20260926 01:10:10.669314 3414 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384993574762-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384993574762-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384993574762-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:10:10.680754 3414 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:10:10.682711 3414 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:10:10.682823 3414 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:10:10.686532 3414 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38895
I20260926 01:10:10.686570 3481 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38895 every 8 connection(s)
I20260926 01:10:10.686908 3414 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
I20260926 01:10:10.687968 3482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:10:10.690320 3482 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f86b7c9fbe5429eb7a4b2e26b576281: Bootstrap starting.
I20260926 01:10:10.690881 3482 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9f86b7c9fbe5429eb7a4b2e26b576281: Neither blocks nor log segments found. Creating new log.
I20260926 01:10:10.691129 3482 log.cc:824] T 00000000000000000000000000000000 P 9f86b7c9fbe5429eb7a4b2e26b576281: Log is configured to *not* fsync() on all Append() calls
I20260926 01:10:10.691506 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 3414
I20260926 01:10:10.691661 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384993574762-392-0/minicluster-data/master-0/wal/instance
I20260926 01:10:10.691850 392 external_mini_cluster.cc:1428] Setting key 13dd0844c2e3fc86be0f4c207bd21b36
I20260926 01:10:10.691893 3482 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f86b7c9fbe5429eb7a4b2e26b576281: No bootstrap required, opened a new log
I20260926 01:10:10.694232 3482 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9f86b7c9fbe5429eb7a4b2e26b576281 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f86b7c9fbe5429eb7a4b2e26b576281" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38895 } }
I20260926 01:10:10.694375 3482 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9f86b7c9fbe5429eb7a4b2e26b576281 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:10:10.694396 3482 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9f86b7c9fbe5429eb7a4b2e26b576281 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f86b7c9fbe5429eb7a4b2e26b576281, State: Initialized, Role: FOLLOWER
I20260926 01:10:10.694506 3482 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9f86b7c9fbe5429eb7a4b2e26b576281 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f86b7c9fbe5429eb7a4b2e26b576281" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38895 } }
I20260926 01:10:10.694572 3482 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9f86b7c9fbe5429eb7a4b2e26b576281 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:10:10.694595 3482 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9f86b7c9fbe5429eb7a4b2e26b576281 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:10:10.694649 3482 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9f86b7c9fbe5429eb7a4b2e26b576281 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:10:10.695219 3482 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9f86b7c9fbe5429eb7a4b2e26b576281 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f86b7c9fbe5429eb7a4b2e26b576281" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38895 } }
I20260926 01:10:10.695307 3482 leader_election.cc:302] T 00000000000000000000000000000000 P 9f86b7c9fbe5429eb7a4b2e26b576281 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9f86b7c9fbe5429eb7a4b2e26b576281; no voters:
I20260926 01:10:10.695474 3482 leader_election.cc:288] T 00000000000000000000000000000000 P 9f86b7c9fbe5429eb7a4b2e26b576281 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:10:10.695513 3487 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9f86b7c9fbe5429eb7a4b2e26b576281 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:10:10.695662 3487 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9f86b7c9fbe5429eb7a4b2e26b576281 [term 1 LEADER]: Becoming Leader. State: Replica: 9f86b7c9fbe5429eb7a4b2e26b576281, State: Running, Role: LEADER
I20260926 01:10:10.695744 3482 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9f86b7c9fbe5429eb7a4b2e26b576281 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:10:10.695785 3487 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9f86b7c9fbe5429eb7a4b2e26b576281 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f86b7c9fbe5429eb7a4b2e26b576281" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38895 } }
I20260926 01:10:10.696187 3489 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f86b7c9fbe5429eb7a4b2e26b576281 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9f86b7c9fbe5429eb7a4b2e26b576281" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f86b7c9fbe5429eb7a4b2e26b576281" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38895 } } }
I20260926 01:10:10.696266 3489 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f86b7c9fbe5429eb7a4b2e26b576281 [sys.catalog]: This master's current role is: LEADER
I20260926 01:10:10.696177 3487 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f86b7c9fbe5429eb7a4b2e26b576281 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f86b7c9fbe5429eb7a4b2e26b576281. Latest consensus state: current_term: 1 leader_uuid: "9f86b7c9fbe5429eb7a4b2e26b576281" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f86b7c9fbe5429eb7a4b2e26b576281" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38895 } } }
I20260926 01:10:10.696529 3487 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f86b7c9fbe5429eb7a4b2e26b576281 [sys.catalog]: This master's current role is: LEADER
I20260926 01:10:10.696861 3495 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:10:10.697393 3495 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:10:10.699163 3495 catalog_manager.cc:1347] Generated new cluster ID: 60a6428dc49d4924a200330c240ee3d7
I20260926 01:10:10.699209 3495 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 01:10:10 dist-test-slave-7b22 krb5kdc[3401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385010, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:10:10.703552 3486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:10.692364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57798 (local address 127.0.98.62:38895)
0926 01:10:10.692526 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:10.692536 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:10.692562 (+ 26us) server_negotiation.cc:407] Connection header received
0926 01:10:10.693160 (+ 598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:10.693168 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:10.693221 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:10.693338 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:10.694121 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:10.695382 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:10.696455 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:10.697117 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:10.697209 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:10.700045 (+ 2836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:10:10.700049 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:10:10.700050 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:10:10.700062 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:10:10.701970 (+ 1908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:10.702380 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:10.702382 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:10.702383 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:10.702480 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:10.702675 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:10.702676 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:10.702676 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:10.702963 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:10:10.703109 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:10.703222 (+ 113us) server_negotiation.cc:299] Negotiation successful
0926 01:10:10.703352 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":60,"threads_started":1}
I20260926 01:10:10.706115 3495 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:10:10.706614 3495 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:10:10.710142 3495 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9f86b7c9fbe5429eb7a4b2e26b576281: Generated new TSK 0
I20260926 01:10:10.710327 3495 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384993574762-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384993574762-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 01:10:10.744807 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384993574762-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384993574762-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384993574762-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384993574762-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38895
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384993574762-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:10:10.827598 3508 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:10:10.827831 3508 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:10:10.827865 3508 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:10:10.829105 3508 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:10:10.829149 3508 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:10:10.829188 3508 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 01:10:10.830730 3508 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384993574762-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384993574762-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384993574762-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384993574762-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38895
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3508
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384993574762-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:10:10.830986 3508 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:10:10.831200 3508 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:10:10.836129 3516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:10:10.836102 3513 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:10:10.836114 3514 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:10:10.836228 3508 server_base.cc:1034] running on GCE node
I20260926 01:10:10.836498 3508 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:10:10.836750 3508 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:10:10.837867 3508 hybrid_clock.cc:648] HybridClock initialized: now 1790385010837853 us; error 30 us; skew 500 ppm
Sep 26 01:10:10 dist-test-slave-7b22 krb5kdc[3401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385010, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:10:10.840385 3508 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 01:10:10.840819 3508 webserver.cc:460] Webserver started at http://127.0.98.1:42681/ using document root <none> and password file <none>
I20260926 01:10:10.841035 3508 fs_manager.cc:359] Metadata directory not provided
I20260926 01:10:10.841079 3508 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:10:10.841209 3508 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:10:10.842315 3508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384993574762-392-0/minicluster-data/ts-0/data/instance:
uuid: "08a84339f7d749afa9c0f6cfe0801649"
format_stamp: "Formatted at 2026-09-26 01:10:10 on dist-test-slave-7b22"
server_key: "81af4f3f7cc5048b2d58f5e34691f9ab"
server_key_iv: "bea4c5cbcad1e200f854c3261bd833f0"
server_key_version: "encryptionkey@0"
I20260926 01:10:10.842643 3508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384993574762-392-0/minicluster-data/ts-0/wal/instance:
uuid: "08a84339f7d749afa9c0f6cfe0801649"
format_stamp: "Formatted at 2026-09-26 01:10:10 on dist-test-slave-7b22"
server_key: "81af4f3f7cc5048b2d58f5e34691f9ab"
server_key_iv: "bea4c5cbcad1e200f854c3261bd833f0"
server_key_version: "encryptionkey@0"
I20260926 01:10:10.843953 3508 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 01:10:10.844702 3523 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:10.844854 3508 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 01:10:10.844930 3508 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384993574762-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384993574762-392-0/minicluster-data/ts-0/wal
uuid: "08a84339f7d749afa9c0f6cfe0801649"
format_stamp: "Formatted at 2026-09-26 01:10:10 on dist-test-slave-7b22"
server_key: "81af4f3f7cc5048b2d58f5e34691f9ab"
server_key_iv: "bea4c5cbcad1e200f854c3261bd833f0"
server_key_version: "encryptionkey@0"
I20260926 01:10:10.844998 3508 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384993574762-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384993574762-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384993574762-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:10:10.878716 3508 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:10:10.880772 3508 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:10:10.880904 3508 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:10:10.881146 3508 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:10:10.881467 3508 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:10:10.881502 3508 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:10.881525 3508 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:10:10.881548 3508 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:10.887168 3508 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41311
I20260926 01:10:10.887226 3636 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41311 every 8 connection(s)
I20260926 01:10:10.887550 3508 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384993574762-392-0/minicluster-data/ts-0/data/info.pb
I20260926 01:10:10.890334 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 3508
I20260926 01:10:10.890425 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384993574762-392-0/minicluster-data/ts-0/wal/instance
I20260926 01:10:10.890599 392 external_mini_cluster.cc:1428] Setting key ab85651556ef2ea10772dfc96cbbd381
Sep 26 01:10:10 dist-test-slave-7b22 krb5kdc[3401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385010, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:10:10.898394 3486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:10.888301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43725 (local address 127.0.98.62:38895)
0926 01:10:10.888365 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:10.888367 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:10.888585 (+ 218us) server_negotiation.cc:407] Connection header received
0926 01:10:10.889107 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:10.889109 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:10.889119 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:10.889185 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:10.890218 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:10.890829 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:10.892237 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:10.892516 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:10.892590 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:10.895100 (+ 2510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:10:10.895102 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:10:10.895102 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:10:10.895110 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:10:10.896875 (+ 1765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:10.897328 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:10.897329 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:10.897329 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:10.897397 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:10.897637 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:10.897638 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:10.897639 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:10.897791 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:10:10.897893 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:10.898201 (+ 308us) server_negotiation.cc:299] Negotiation successful
0926 01:10:10.898293 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 01:10:10.898504 3639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:10.888284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38895 (local address 127.0.98.1:43725)
0926 01:10:10.888463 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:10.888502 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:10.889022 (+ 520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:10.889236 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:10.889240 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:10.889878 (+ 638us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:10:10.890103 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:10.890109 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:10.890937 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:10.890938 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:10.892124 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:10.892128 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:10.892607 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:10.892608 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:10.892751 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:10.893278 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:10:10.893287 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:10:10.894972 (+ 1685us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:10:10.896960 (+ 1988us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:10:10.896968 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:10:10.896970 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:10:10.897249 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:10:10.897461 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:10:10.897461 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:10:10.897462 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:10:10.897579 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:10:10.897905 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:10:10.897913 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:10:10.898058 (+ 145us) client_negotiation.cc:769] Sending connection context
0926 01:10:10.898162 (+ 104us) client_negotiation.cc:240] Negotiation successful
0926 01:10:10.898308 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":101,"threads_started":1}
I20260926 01:10:10.899170 3637 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38895
I20260926 01:10:10.899262 3637 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260926 01:10:10.899457 3637 heartbeater.cc:511] Master 127.0.98.62:38895 requested a full tablet report, sending...
I20260926 01:10:10.900039 3446 ts_manager.cc:194] Registered new tserver with Master: 08a84339f7d749afa9c0f6cfe0801649 (127.0.98.1:41311)
I20260926 01:10:10.900799 3446 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:43725
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384993574762-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384993574762-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 01:10:10.918223 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384993574762-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384993574762-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384993574762-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384993574762-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38895
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384993574762-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:10:11.008810 3642 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:10:11.009053 3642 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:10:11.009085 3642 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:10:11.010818 3642 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:10:11.010885 3642 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:10:11.010917 3642 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 01:10:11.012619 3642 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384993574762-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384993574762-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384993574762-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384993574762-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38895
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3642
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384993574762-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:10:11.012885 3642 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:10:11.013123 3642 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 01:10:11.018291 3642 server_base.cc:1034] running on GCE node
W20260926 01:10:11.018277 3650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:10:11.018420 3648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:10:11.018271 3647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:10:11.018723 3642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:10:11.018970 3642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:10:11.020136 3642 hybrid_clock.cc:648] HybridClock initialized: now 1790385011020116 us; error 34 us; skew 500 ppm
Sep 26 01:10:11 dist-test-slave-7b22 krb5kdc[3401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385011, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:10:11.022789 3642 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 01:10:11.023332 3642 webserver.cc:460] Webserver started at http://127.0.98.2:41415/ using document root <none> and password file <none>
I20260926 01:10:11.023571 3642 fs_manager.cc:359] Metadata directory not provided
I20260926 01:10:11.023617 3642 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:10:11.023733 3642 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:10:11.025025 3642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384993574762-392-0/minicluster-data/ts-1/data/instance:
uuid: "cf4c3c47b53244d0ae7e10f9f5c8427f"
format_stamp: "Formatted at 2026-09-26 01:10:11 on dist-test-slave-7b22"
server_key: "3e7eafa68c3ecf038e8f78e2c0f2399f"
server_key_iv: "527c7fc9fe9c62a62bcc01c040e6942c"
server_key_version: "encryptionkey@0"
I20260926 01:10:11.025442 3642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384993574762-392-0/minicluster-data/ts-1/wal/instance:
uuid: "cf4c3c47b53244d0ae7e10f9f5c8427f"
format_stamp: "Formatted at 2026-09-26 01:10:11 on dist-test-slave-7b22"
server_key: "3e7eafa68c3ecf038e8f78e2c0f2399f"
server_key_iv: "527c7fc9fe9c62a62bcc01c040e6942c"
server_key_version: "encryptionkey@0"
I20260926 01:10:11.026998 3642 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 01:10:11.027892 3657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:11.028090 3642 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 01:10:11.028210 3642 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384993574762-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384993574762-392-0/minicluster-data/ts-1/wal
uuid: "cf4c3c47b53244d0ae7e10f9f5c8427f"
format_stamp: "Formatted at 2026-09-26 01:10:11 on dist-test-slave-7b22"
server_key: "3e7eafa68c3ecf038e8f78e2c0f2399f"
server_key_iv: "527c7fc9fe9c62a62bcc01c040e6942c"
server_key_version: "encryptionkey@0"
I20260926 01:10:11.028280 3642 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384993574762-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384993574762-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384993574762-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:10:11.047564 3642 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:10:11.049968 3642 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:10:11.050115 3642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:10:11.050349 3642 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:10:11.050675 3642 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:10:11.050707 3642 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:11.050731 3642 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:10:11.050756 3642 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:11.056393 3642 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44085
I20260926 01:10:11.056453 3770 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44085 every 8 connection(s)
I20260926 01:10:11.056753 3642 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384993574762-392-0/minicluster-data/ts-1/data/info.pb
Sep 26 01:10:11 dist-test-slave-7b22 krb5kdc[3401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385011, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:10:11.064003 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 3642
I20260926 01:10:11.064129 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384993574762-392-0/minicluster-data/ts-1/wal/instance
I20260926 01:10:11.064321 392 external_mini_cluster.cc:1428] Setting key 1454858ca614e529a4a552c8ead813b5
I20260926 01:10:11.066533 3486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:11.057562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33837 (local address 127.0.98.62:38895)
0926 01:10:11.057617 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:11.057620 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:11.058051 (+ 431us) server_negotiation.cc:407] Connection header received
0926 01:10:11.058458 (+ 407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:11.058459 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:11.058484 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:11.058532 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:11.059568 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:11.060069 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:11.061129 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:11.061338 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:11.061395 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:11.063567 (+ 2172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:10:11.063569 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:10:11.063569 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:10:11.063577 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:10:11.065143 (+ 1566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:11.065530 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:11.065532 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:11.065532 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:11.065590 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:11.065882 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:11.065883 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:11.065884 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:11.066008 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:10:11.066063 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:11.066351 (+ 288us) server_negotiation.cc:299] Negotiation successful
0926 01:10:11.066444 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260926 01:10:11.066722 3773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:11.057522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38895 (local address 127.0.98.2:33837)
0926 01:10:11.057737 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:11.057781 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:11.058359 (+ 578us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:11.058612 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:11.058616 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:11.059264 (+ 648us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:10:11.059470 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:11.059475 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:11.060156 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:11.060157 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:11.061036 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:11.061038 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:11.061413 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:11.061414 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:11.061540 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:11.062090 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:10:11.062099 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:10:11.063472 (+ 1373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:10:11.065220 (+ 1748us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:10:11.065228 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:10:11.065230 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:10:11.065463 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:10:11.065643 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:10:11.065644 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:10:11.065644 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:10:11.065806 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:10:11.066061 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:10:11.066068 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:10:11.066217 (+ 149us) client_negotiation.cc:769] Sending connection context
0926 01:10:11.066302 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 01:10:11.066485 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":71,"threads_started":1}
I20260926 01:10:11.067210 3771 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38895
I20260926 01:10:11.067312 3771 heartbeater.cc:464] Registering TS with master...
I20260926 01:10:11.067494 3771 heartbeater.cc:511] Master 127.0.98.62:38895 requested a full tablet report, sending...
I20260926 01:10:11.067867 3446 ts_manager.cc:194] Registered new tserver with Master: cf4c3c47b53244d0ae7e10f9f5c8427f (127.0.98.2:44085)
I20260926 01:10:11.068246 3446 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:33837
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384993574762-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384993574762-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 01:10:11.093670 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384993574762-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384993574762-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384993574762-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384993574762-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38895
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384993574762-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:10:11.174762 3776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:10:11.174974 3776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:10:11.174997 3776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:10:11.176157 3776 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:10:11.176196 3776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:10:11.176221 3776 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 01:10:11.177644 3776 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384993574762-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384993574762-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384993574762-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384993574762-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38895
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3776
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384993574762-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:10:11.177881 3776 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:10:11.178108 3776 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 01:10:11.183087 3776 server_base.cc:1034] running on GCE node
W20260926 01:10:11.183048 3782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:10:11.183044 3784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:10:11.183457 3781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:10:11.183777 3776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:10:11.184051 3776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:10:11.185217 3776 hybrid_clock.cc:648] HybridClock initialized: now 1790385011185204 us; error 31 us; skew 500 ppm
Sep 26 01:10:11 dist-test-slave-7b22 krb5kdc[3401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385011, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:10:11.188000 3776 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 01:10:11.188519 3776 webserver.cc:460] Webserver started at http://127.0.98.3:45443/ using document root <none> and password file <none>
I20260926 01:10:11.188721 3776 fs_manager.cc:359] Metadata directory not provided
I20260926 01:10:11.188764 3776 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:10:11.188860 3776 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:10:11.189966 3776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384993574762-392-0/minicluster-data/ts-2/data/instance:
uuid: "bcfbfbdcea4e42f2a693f078589510e4"
format_stamp: "Formatted at 2026-09-26 01:10:11 on dist-test-slave-7b22"
server_key: "097a90cb3a67b8415f8f3bf6f4f82ead"
server_key_iv: "35ee39ac2b58a825b4671bbc5766b6dd"
server_key_version: "encryptionkey@0"
I20260926 01:10:11.190286 3776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384993574762-392-0/minicluster-data/ts-2/wal/instance:
uuid: "bcfbfbdcea4e42f2a693f078589510e4"
format_stamp: "Formatted at 2026-09-26 01:10:11 on dist-test-slave-7b22"
server_key: "097a90cb3a67b8415f8f3bf6f4f82ead"
server_key_iv: "35ee39ac2b58a825b4671bbc5766b6dd"
server_key_version: "encryptionkey@0"
I20260926 01:10:11.191481 3776 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 01:10:11.192188 3791 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:11.192355 3776 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 01:10:11.192425 3776 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384993574762-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384993574762-392-0/minicluster-data/ts-2/wal
uuid: "bcfbfbdcea4e42f2a693f078589510e4"
format_stamp: "Formatted at 2026-09-26 01:10:11 on dist-test-slave-7b22"
server_key: "097a90cb3a67b8415f8f3bf6f4f82ead"
server_key_iv: "35ee39ac2b58a825b4671bbc5766b6dd"
server_key_version: "encryptionkey@0"
I20260926 01:10:11.192476 3776 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384993574762-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384993574762-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384993574762-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:10:11.224767 3776 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:10:11.227052 3776 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:10:11.227173 3776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:10:11.227363 3776 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:10:11.227630 3776 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:10:11.227660 3776 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:11.227682 3776 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:10:11.227694 3776 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:11.233335 3776 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41969
I20260926 01:10:11.233376 3904 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41969 every 8 connection(s)
I20260926 01:10:11.233701 3776 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384993574762-392-0/minicluster-data/ts-2/data/info.pb
I20260926 01:10:11.238879 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 3776
I20260926 01:10:11.238968 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384993574762-392-0/minicluster-data/ts-2/wal/instance
I20260926 01:10:11.239125 392 external_mini_cluster.cc:1428] Setting key 2350bae1104d926b75a511dcded20487
Sep 26 01:10:11 dist-test-slave-7b22 krb5kdc[3401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385011, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:10:11.243244 3486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:11.234399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33447 (local address 127.0.98.62:38895)
0926 01:10:11.234457 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:11.234459 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:11.234692 (+ 233us) server_negotiation.cc:407] Connection header received
0926 01:10:11.235228 (+ 536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:11.235230 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:11.235241 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:11.235296 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:11.236258 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:11.236847 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:11.237885 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:11.238090 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:11.238150 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:11.240576 (+ 2426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:10:11.240577 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:10:11.240577 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:10:11.240584 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:10:11.242008 (+ 1424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:11.242387 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:11.242388 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:11.242388 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:11.242440 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:11.242629 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:11.242630 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:11.242630 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:11.242740 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:10:11.242820 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:11.243083 (+ 263us) server_negotiation.cc:299] Negotiation successful
0926 01:10:11.243166 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260926 01:10:11.243394 3907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:11.234395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38895 (local address 127.0.98.3:33447)
0926 01:10:11.234566 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:11.234605 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:11.235093 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:11.235365 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:11.235368 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:11.235940 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:10:11.236157 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:11.236162 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:11.236927 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:11.236928 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:11.237770 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:11.237772 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:11.238165 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:11.238166 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:11.238273 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:11.238851 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:10:11.238859 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:10:11.240476 (+ 1617us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:10:11.242080 (+ 1604us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:10:11.242086 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:10:11.242088 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:10:11.242324 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:10:11.242486 (+ 162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:10:11.242487 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:10:11.242487 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:10:11.242583 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:10:11.242812 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:10:11.242815 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:10:11.242950 (+ 135us) client_negotiation.cc:769] Sending connection context
0926 01:10:11.243032 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 01:10:11.243180 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":103,"threads_started":1}
I20260926 01:10:11.243994 3905 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38895
I20260926 01:10:11.244086 3905 heartbeater.cc:464] Registering TS with master...
I20260926 01:10:11.244256 3905 heartbeater.cc:511] Master 127.0.98.62:38895 requested a full tablet report, sending...
I20260926 01:10:11.244652 3446 ts_manager.cc:194] Registered new tserver with Master: bcfbfbdcea4e42f2a693f078589510e4 (127.0.98.3:41969)
I20260926 01:10:11.245010 3446 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:33447
I20260926 01:10:11.251034 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 01:10:11 dist-test-slave-7b22 krb5kdc[3401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385011, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 26 01:10:11 dist-test-slave-7b22 krb5kdc[3401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385011, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:10:11.273794 3486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:11.266647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57804 (local address 127.0.98.62:38895)
0926 01:10:11.266752 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:11.266754 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:11.266849 (+ 95us) server_negotiation.cc:407] Connection header received
0926 01:10:11.266914 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:11.266915 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:11.266925 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:11.266974 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:11.267591 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:11.268083 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:11.268821 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:11.269005 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:11.269060 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:11.271300 (+ 2240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:10:11.271302 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:10:11.271302 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:10:11.271310 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:10:11.272646 (+ 1336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:11.273016 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:11.273017 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:11.273018 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:11.273076 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:11.273246 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:11.273247 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:11.273247 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:11.273398 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:10:11.273492 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:11.273615 (+ 123us) server_negotiation.cc:299] Negotiation successful
0926 01:10:11.273703 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
W20260926 01:10:11.274036 3446 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:57804
I20260926 01:10:11.274449 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 3508
I20260926 01:10:11.278705 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 3642
I20260926 01:10:11.282502 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 3776
I20260926 01:10:11.286085 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 3414
2026-09-26T01:10:11Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1793 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384993574762-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 01:10:11 dist-test-slave-7b22 krb5kdc[3917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 01:10:11 dist-test-slave-7b22 krb5kdc[3917](info): set up 2 sockets
Sep 26 01:10:11 dist-test-slave-7b22 krb5kdc[3917](info): commencing operation
krb5kdc: starting...
W20260926 01:10:12.249616 392 mini_kdc.cc:121] Time spent starting KDC: real 0.945s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384993574762-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384993574762-392-0/krb5kdc/test-user.keytab.
Sep 26 01:10:12 dist-test-slave-7b22 krb5kdc[3917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385012, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T01:10:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:10:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384993574762-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384993574762-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 01:10:12.369567 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384993574762-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384993574762-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384993574762-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42373
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384993574762-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42373
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:10:12.451857 3930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:10:12.452148 3930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:10:12.452185 3930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:10:12.453584 3930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:10:12.453644 3930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:10:12.453665 3930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:10:12.453680 3930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:10:12.453696 3930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:10:12.455443 3930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384993574762-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384993574762-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42373
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384993574762-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42373
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384993574762-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:10:12.455742 3930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:10:12.455976 3930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:10:12.461025 3938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:10:12.460973 3930 server_base.cc:1034] running on GCE node
W20260926 01:10:12.461018 3936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:10:12.460973 3935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:10:12.461396 3930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:10:12.461696 3930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:10:12.462875 3930 hybrid_clock.cc:648] HybridClock initialized: now 1790385012462825 us; error 84 us; skew 500 ppm
Sep 26 01:10:12 dist-test-slave-7b22 krb5kdc[3917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385012, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:10:12.465631 3930 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 01:10:12.466214 3930 webserver.cc:460] Webserver started at http://127.0.98.62:46349/ using document root <none> and password file <none>
I20260926 01:10:12.466434 3930 fs_manager.cc:359] Metadata directory not provided
I20260926 01:10:12.466504 3930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:10:12.466646 3930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:10:12.467767 3930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384993574762-392-0/minicluster-data/master-0/data/instance:
uuid: "ec69a939bbac46eb987a67a95dfca61d"
format_stamp: "Formatted at 2026-09-26 01:10:12 on dist-test-slave-7b22"
server_key: "e232beb1d2578091d5b9b74e41ece4cc"
server_key_iv: "918d8fea3c54b6407377aecdcddad71f"
server_key_version: "encryptionkey@0"
I20260926 01:10:12.468152 3930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384993574762-392-0/minicluster-data/master-0/wal/instance:
uuid: "ec69a939bbac46eb987a67a95dfca61d"
format_stamp: "Formatted at 2026-09-26 01:10:12 on dist-test-slave-7b22"
server_key: "e232beb1d2578091d5b9b74e41ece4cc"
server_key_iv: "918d8fea3c54b6407377aecdcddad71f"
server_key_version: "encryptionkey@0"
I20260926 01:10:12.469540 3930 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 01:10:12.470376 3945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:12.470537 3930 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:12.470629 3930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384993574762-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384993574762-392-0/minicluster-data/master-0/wal
uuid: "ec69a939bbac46eb987a67a95dfca61d"
format_stamp: "Formatted at 2026-09-26 01:10:12 on dist-test-slave-7b22"
server_key: "e232beb1d2578091d5b9b74e41ece4cc"
server_key_iv: "918d8fea3c54b6407377aecdcddad71f"
server_key_version: "encryptionkey@0"
I20260926 01:10:12.470706 3930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384993574762-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384993574762-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384993574762-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:10:12.481500 3930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:10:12.483934 3930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:10:12.484122 3930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:10:12.488240 3930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42373
I20260926 01:10:12.488284 3997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42373 every 8 connection(s)
I20260926 01:10:12.488654 3930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
I20260926 01:10:12.489545 3998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:10:12.491945 3998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec69a939bbac46eb987a67a95dfca61d: Bootstrap starting.
I20260926 01:10:12.492478 3998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ec69a939bbac46eb987a67a95dfca61d: Neither blocks nor log segments found. Creating new log.
I20260926 01:10:12.492719 3998 log.cc:824] T 00000000000000000000000000000000 P ec69a939bbac46eb987a67a95dfca61d: Log is configured to *not* fsync() on all Append() calls
I20260926 01:10:12.493420 3998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec69a939bbac46eb987a67a95dfca61d: No bootstrap required, opened a new log
I20260926 01:10:12.494658 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 3930
I20260926 01:10:12.494753 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384993574762-392-0/minicluster-data/master-0/wal/instance
I20260926 01:10:12.494951 392 external_mini_cluster.cc:1428] Setting key c818949bf87daabbff939d646bc6cee6
I20260926 01:10:12.495306 3998 raft_consensus.cc:348] T 00000000000000000000000000000000 P ec69a939bbac46eb987a67a95dfca61d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec69a939bbac46eb987a67a95dfca61d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42373 } }
I20260926 01:10:12.495498 3998 raft_consensus.cc:374] T 00000000000000000000000000000000 P ec69a939bbac46eb987a67a95dfca61d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:10:12.495539 3998 raft_consensus.cc:729] T 00000000000000000000000000000000 P ec69a939bbac46eb987a67a95dfca61d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec69a939bbac46eb987a67a95dfca61d, State: Initialized, Role: FOLLOWER
I20260926 01:10:12.495676 3998 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec69a939bbac46eb987a67a95dfca61d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec69a939bbac46eb987a67a95dfca61d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42373 } }
I20260926 01:10:12.495755 3998 raft_consensus.cc:388] T 00000000000000000000000000000000 P ec69a939bbac46eb987a67a95dfca61d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:10:12.495786 3998 raft_consensus.cc:482] T 00000000000000000000000000000000 P ec69a939bbac46eb987a67a95dfca61d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:10:12.495831 3998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ec69a939bbac46eb987a67a95dfca61d [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:10:12.496927 3998 raft_consensus.cc:504] T 00000000000000000000000000000000 P ec69a939bbac46eb987a67a95dfca61d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec69a939bbac46eb987a67a95dfca61d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42373 } }
I20260926 01:10:12.497046 3998 leader_election.cc:302] T 00000000000000000000000000000000 P ec69a939bbac46eb987a67a95dfca61d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ec69a939bbac46eb987a67a95dfca61d; no voters:
I20260926 01:10:12.497219 3998 leader_election.cc:288] T 00000000000000000000000000000000 P ec69a939bbac46eb987a67a95dfca61d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:10:12.497267 4003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ec69a939bbac46eb987a67a95dfca61d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:10:12.497442 4003 raft_consensus.cc:686] T 00000000000000000000000000000000 P ec69a939bbac46eb987a67a95dfca61d [term 1 LEADER]: Becoming Leader. State: Replica: ec69a939bbac46eb987a67a95dfca61d, State: Running, Role: LEADER
I20260926 01:10:12.497484 3998 sys_catalog.cc:564] T 00000000000000000000000000000000 P ec69a939bbac46eb987a67a95dfca61d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:10:12.497538 4003 consensus_queue.cc:237] T 00000000000000000000000000000000 P ec69a939bbac46eb987a67a95dfca61d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec69a939bbac46eb987a67a95dfca61d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42373 } }
I20260926 01:10:12.497880 4003 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec69a939bbac46eb987a67a95dfca61d [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec69a939bbac46eb987a67a95dfca61d. Latest consensus state: current_term: 1 leader_uuid: "ec69a939bbac46eb987a67a95dfca61d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec69a939bbac46eb987a67a95dfca61d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42373 } } }
I20260926 01:10:12.497954 4003 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec69a939bbac46eb987a67a95dfca61d [sys.catalog]: This master's current role is: LEADER
I20260926 01:10:12.497907 4005 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec69a939bbac46eb987a67a95dfca61d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ec69a939bbac46eb987a67a95dfca61d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec69a939bbac46eb987a67a95dfca61d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42373 } } }
I20260926 01:10:12.498153 4009 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:10:12.498167 4005 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec69a939bbac46eb987a67a95dfca61d [sys.catalog]: This master's current role is: LEADER
I20260926 01:10:12.498682 4009 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:10:12.501946 4009 catalog_manager.cc:1347] Generated new cluster ID: 3d9f3e69215a40edb05651e37eb3ea8f
I20260926 01:10:12.502003 4009 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 01:10:12 dist-test-slave-7b22 krb5kdc[3917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385012, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:10:12.511173 4002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:12.495490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54668 (local address 127.0.98.62:42373)
0926 01:10:12.499426 (+ 3936us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:12.499437 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:12.499643 (+ 206us) server_negotiation.cc:407] Connection header received
0926 01:10:12.499898 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:12.499907 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:12.499971 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:12.500092 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:12.501122 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:12.502557 (+ 1435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:12.503645 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:12.504375 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:12.504454 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:12.507426 (+ 2972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:10:12.507430 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:10:12.507431 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:10:12.507445 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:10:12.509426 (+ 1981us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:12.509901 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:12.509904 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:12.509905 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:12.510024 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:12.510230 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:12.510230 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:12.510231 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:12.510539 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:10:12.510692 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:12.510787 (+ 95us) server_negotiation.cc:299] Negotiation successful
0926 01:10:12.510926 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3880,"thread_start_us":80,"threads_started":1}
I20260926 01:10:12.529371 4009 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:10:12.530169 4009 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:10:12.546224 4009 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ec69a939bbac46eb987a67a95dfca61d: Generated new TSK 0
I20260926 01:10:12.546442 4009 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384993574762-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384993574762-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 01:10:12.577631 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384993574762-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384993574762-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384993574762-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384993574762-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42373
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384993574762-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:10:12.660224 4024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:10:12.660504 4024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:10:12.660539 4024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:10:12.661783 4024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:10:12.661856 4024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:10:12.661895 4024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 01:10:12.663467 4024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384993574762-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384993574762-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384993574762-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384993574762-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42373
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384993574762-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:10:12.663671 4024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:10:12.663859 4024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:10:12.668742 4032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:10:12.668744 4029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:10:12.668744 4030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:10:12.668990 4024 server_base.cc:1034] running on GCE node
I20260926 01:10:12.669240 4024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:10:12.669483 4024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:10:12.670621 4024 hybrid_clock.cc:648] HybridClock initialized: now 1790385012670622 us; error 55 us; skew 500 ppm
Sep 26 01:10:12 dist-test-slave-7b22 krb5kdc[3917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385012, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:10:12.673158 4024 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 01:10:12.673645 4024 webserver.cc:460] Webserver started at http://127.0.98.1:42399/ using document root <none> and password file <none>
I20260926 01:10:12.673906 4024 fs_manager.cc:359] Metadata directory not provided
I20260926 01:10:12.673959 4024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:10:12.674086 4024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:10:12.675169 4024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384993574762-392-0/minicluster-data/ts-0/data/instance:
uuid: "b059c8adce664edda5078b486752d75a"
format_stamp: "Formatted at 2026-09-26 01:10:12 on dist-test-slave-7b22"
server_key: "5103734f22b8c63902101a1adc5db5d4"
server_key_iv: "68194ac6b7075c5311039290621f0038"
server_key_version: "encryptionkey@0"
I20260926 01:10:12.675508 4024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384993574762-392-0/minicluster-data/ts-0/wal/instance:
uuid: "b059c8adce664edda5078b486752d75a"
format_stamp: "Formatted at 2026-09-26 01:10:12 on dist-test-slave-7b22"
server_key: "5103734f22b8c63902101a1adc5db5d4"
server_key_iv: "68194ac6b7075c5311039290621f0038"
server_key_version: "encryptionkey@0"
I20260926 01:10:12.676826 4024 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 01:10:12.677577 4039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:12.677855 4024 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:10:12.677958 4024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384993574762-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384993574762-392-0/minicluster-data/ts-0/wal
uuid: "b059c8adce664edda5078b486752d75a"
format_stamp: "Formatted at 2026-09-26 01:10:12 on dist-test-slave-7b22"
server_key: "5103734f22b8c63902101a1adc5db5d4"
server_key_iv: "68194ac6b7075c5311039290621f0038"
server_key_version: "encryptionkey@0"
I20260926 01:10:12.678030 4024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384993574762-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384993574762-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384993574762-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:10:12.691712 4024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:10:12.693738 4024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:10:12.693921 4024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:10:12.694146 4024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:10:12.694448 4024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:10:12.694487 4024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:12.694517 4024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:10:12.694540 4024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:12.700138 4024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36929
I20260926 01:10:12.700206 4152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36929 every 8 connection(s)
I20260926 01:10:12.700501 4024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384993574762-392-0/minicluster-data/ts-0/data/info.pb
I20260926 01:10:12.702561 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 4024
I20260926 01:10:12.702716 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384993574762-392-0/minicluster-data/ts-0/wal/instance
I20260926 01:10:12.702914 392 external_mini_cluster.cc:1428] Setting key 7b2959650892ec13283a3030f6779ffe
Sep 26 01:10:12 dist-test-slave-7b22 krb5kdc[3917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385012, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260926 01:10:12.711061 4002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:12.701232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35623 (local address 127.0.98.62:42373)
0926 01:10:12.701304 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:12.701306 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:12.701498 (+ 192us) server_negotiation.cc:407] Connection header received
0926 01:10:12.702537 (+ 1039us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:12.702539 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:12.702554 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:12.702633 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:12.703397 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:12.703966 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:12.705271 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:12.705568 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:12.705641 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:12.707950 (+ 2309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:10:12.707952 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:10:12.707952 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:10:12.707960 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:10:12.709496 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:12.709964 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:12.709966 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:12.709966 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:12.710030 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:12.710313 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:12.710315 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:12.710315 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:12.710457 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:10:12.710528 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:12.710873 (+ 345us) server_negotiation.cc:299] Negotiation successful
0926 01:10:12.710959 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 01:10:12.711191 4155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:12.701205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42373 (local address 127.0.98.1:35623)
0926 01:10:12.701375 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:12.701412 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:12.702404 (+ 992us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:12.702716 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:12.702721 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:12.703052 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:10:12.703311 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:12.703317 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:12.704068 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:12.704077 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:12.705190 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:12.705193 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:12.705647 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:12.705648 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:12.705750 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:12.706303 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:10:12.706312 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:10:12.707846 (+ 1534us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:10:12.709607 (+ 1761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:10:12.709615 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:10:12.709617 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:10:12.709894 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:10:12.710116 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:10:12.710117 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:10:12.710118 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:10:12.710235 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:10:12.710523 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:10:12.710527 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:10:12.710729 (+ 202us) client_negotiation.cc:769] Sending connection context
0926 01:10:12.710815 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 01:10:12.710953 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":90,"threads_started":1}
I20260926 01:10:12.711781 4153 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42373
I20260926 01:10:12.711882 4153 heartbeater.cc:464] Registering TS with master...
I20260926 01:10:12.712042 4153 heartbeater.cc:511] Master 127.0.98.62:42373 requested a full tablet report, sending...
I20260926 01:10:12.712603 3962 ts_manager.cc:194] Registered new tserver with Master: b059c8adce664edda5078b486752d75a (127.0.98.1:36929)
I20260926 01:10:12.713282 3962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:35623
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384993574762-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384993574762-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 01:10:12.729409 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384993574762-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384993574762-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384993574762-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384993574762-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42373
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384993574762-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:10:12.811357 4158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:10:12.811584 4158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:10:12.811617 4158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:10:12.812811 4158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:10:12.812857 4158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:10:12.812894 4158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 01:10:12.814451 4158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384993574762-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384993574762-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384993574762-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384993574762-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42373
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384993574762-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:10:12.814771 4158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:10:12.814996 4158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:10:12.820097 4164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:10:12.820142 4166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:10:12.820178 4158 server_base.cc:1034] running on GCE node
W20260926 01:10:12.820168 4163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:10:12.820509 4158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:10:12.820765 4158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:10:12.821882 4158 hybrid_clock.cc:648] HybridClock initialized: now 1790385012821861 us; error 38 us; skew 500 ppm
Sep 26 01:10:12 dist-test-slave-7b22 krb5kdc[3917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385012, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:10:12.824492 4158 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 01:10:12.824946 4158 webserver.cc:460] Webserver started at http://127.0.98.2:45811/ using document root <none> and password file <none>
I20260926 01:10:12.825148 4158 fs_manager.cc:359] Metadata directory not provided
I20260926 01:10:12.825205 4158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:10:12.825330 4158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:10:12.826450 4158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384993574762-392-0/minicluster-data/ts-1/data/instance:
uuid: "85d76fe2b8624ab0967667863a284f51"
format_stamp: "Formatted at 2026-09-26 01:10:12 on dist-test-slave-7b22"
server_key: "295bb83732676783117af65cfacaffa3"
server_key_iv: "58a59993d449868f4b485921c7f95f32"
server_key_version: "encryptionkey@0"
I20260926 01:10:12.826793 4158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384993574762-392-0/minicluster-data/ts-1/wal/instance:
uuid: "85d76fe2b8624ab0967667863a284f51"
format_stamp: "Formatted at 2026-09-26 01:10:12 on dist-test-slave-7b22"
server_key: "295bb83732676783117af65cfacaffa3"
server_key_iv: "58a59993d449868f4b485921c7f95f32"
server_key_version: "encryptionkey@0"
I20260926 01:10:12.828212 4158 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 01:10:12.829077 4173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:12.829308 4158 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:10:12.829399 4158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384993574762-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384993574762-392-0/minicluster-data/ts-1/wal
uuid: "85d76fe2b8624ab0967667863a284f51"
format_stamp: "Formatted at 2026-09-26 01:10:12 on dist-test-slave-7b22"
server_key: "295bb83732676783117af65cfacaffa3"
server_key_iv: "58a59993d449868f4b485921c7f95f32"
server_key_version: "encryptionkey@0"
I20260926 01:10:12.829464 4158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384993574762-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384993574762-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384993574762-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:10:12.842041 4158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:10:12.844244 4158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:10:12.844388 4158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:10:12.844624 4158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:10:12.844970 4158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:10:12.845008 4158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:12.845031 4158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:10:12.845042 4158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:12.850414 4158 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34617
I20260926 01:10:12.850479 4286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34617 every 8 connection(s)
I20260926 01:10:12.850778 4158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384993574762-392-0/minicluster-data/ts-1/data/info.pb
I20260926 01:10:12.854417 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 4158
I20260926 01:10:12.854538 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384993574762-392-0/minicluster-data/ts-1/wal/instance
I20260926 01:10:12.854727 392 external_mini_cluster.cc:1428] Setting key 0371921d184d4da93b50dc76d0e0d589
Sep 26 01:10:12 dist-test-slave-7b22 krb5kdc[3917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385012, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:10:12.861168 4002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:12.851471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53567 (local address 127.0.98.62:42373)
0926 01:10:12.851556 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:12.851558 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:12.851806 (+ 248us) server_negotiation.cc:407] Connection header received
0926 01:10:12.852755 (+ 949us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:12.852757 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:12.852768 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:12.852822 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:12.853461 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:12.854007 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:12.855415 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:12.855598 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:12.855651 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:12.858322 (+ 2671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:10:12.858324 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:10:12.858324 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:10:12.858334 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:10:12.859771 (+ 1437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:12.860213 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:12.860215 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:12.860216 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:12.860274 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:12.860512 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:12.860513 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:12.860514 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:12.860633 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:10:12.860693 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:12.860976 (+ 283us) server_negotiation.cc:299] Negotiation successful
0926 01:10:12.861064 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 01:10:12.861349 4289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:12.851446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42373 (local address 127.0.98.2:53567)
0926 01:10:12.851605 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:12.851642 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:12.852627 (+ 985us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:12.852884 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:12.852888 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:12.853154 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:10:12.853357 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:12.853361 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:12.854112 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:12.854113 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:12.855333 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:12.855338 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:12.855675 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:12.855676 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:12.855838 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:12.856405 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:10:12.856416 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:10:12.858195 (+ 1779us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:10:12.859879 (+ 1684us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:10:12.859888 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:10:12.859894 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:10:12.860135 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:10:12.860332 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:10:12.860333 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:10:12.860333 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:10:12.860448 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:10:12.860695 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:10:12.860702 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:10:12.860853 (+ 151us) client_negotiation.cc:769] Sending connection context
0926 01:10:12.860948 (+ 95us) client_negotiation.cc:240] Negotiation successful
0926 01:10:12.861123 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":85,"threads_started":1}
I20260926 01:10:12.861786 4287 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42373
I20260926 01:10:12.861917 4287 heartbeater.cc:464] Registering TS with master...
I20260926 01:10:12.862109 4287 heartbeater.cc:511] Master 127.0.98.62:42373 requested a full tablet report, sending...
I20260926 01:10:12.862506 3962 ts_manager.cc:194] Registered new tserver with Master: 85d76fe2b8624ab0967667863a284f51 (127.0.98.2:34617)
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260926 01:10:12.863029 3962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:53567
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384993574762-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384993574762-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 01:10:12.878942 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384993574762-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384993574762-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384993574762-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384993574762-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42373
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384993574762-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:10:12.961071 4292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:10:12.961292 4292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:10:12.961316 4292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:10:12.962512 4292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:10:12.962554 4292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:10:12.962581 4292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 01:10:12.964123 4292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384993574762-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384993574762-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384993574762-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384993574762-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42373
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384993574762-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:10:12.964324 4292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:10:12.964509 4292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:10:12.969813 4300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:10:12.969858 4298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:10:12.969907 4292 server_base.cc:1034] running on GCE node
W20260926 01:10:12.969813 4297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:10:12.970173 4292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:10:12.970405 4292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:10:12.971570 4292 hybrid_clock.cc:648] HybridClock initialized: now 1790385012971556 us; error 26 us; skew 500 ppm
Sep 26 01:10:12 dist-test-slave-7b22 krb5kdc[3917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385012, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:10:12.974527 4292 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 01:10:12.975028 4292 webserver.cc:460] Webserver started at http://127.0.98.3:46809/ using document root <none> and password file <none>
I20260926 01:10:12.975250 4292 fs_manager.cc:359] Metadata directory not provided
I20260926 01:10:12.975301 4292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:10:12.975427 4292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:10:12.976518 4292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384993574762-392-0/minicluster-data/ts-2/data/instance:
uuid: "e3d486d1f720495b8e1e171a38cc03ad"
format_stamp: "Formatted at 2026-09-26 01:10:12 on dist-test-slave-7b22"
server_key: "924c63513837fb547e720a424b2c81ed"
server_key_iv: "ce494c8276847448101017527c6b6beb"
server_key_version: "encryptionkey@0"
I20260926 01:10:12.976869 4292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384993574762-392-0/minicluster-data/ts-2/wal/instance:
uuid: "e3d486d1f720495b8e1e171a38cc03ad"
format_stamp: "Formatted at 2026-09-26 01:10:12 on dist-test-slave-7b22"
server_key: "924c63513837fb547e720a424b2c81ed"
server_key_iv: "ce494c8276847448101017527c6b6beb"
server_key_version: "encryptionkey@0"
I20260926 01:10:12.978163 4292 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 01:10:12.978920 4307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:12.979125 4292 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 01:10:12.979221 4292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384993574762-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384993574762-392-0/minicluster-data/ts-2/wal
uuid: "e3d486d1f720495b8e1e171a38cc03ad"
format_stamp: "Formatted at 2026-09-26 01:10:12 on dist-test-slave-7b22"
server_key: "924c63513837fb547e720a424b2c81ed"
server_key_iv: "ce494c8276847448101017527c6b6beb"
server_key_version: "encryptionkey@0"
I20260926 01:10:12.979287 4292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384993574762-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384993574762-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384993574762-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:10:13.009729 4292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:10:13.012148 4292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:10:13.012295 4292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:10:13.012537 4292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:10:13.012853 4292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:10:13.012892 4292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:13.012924 4292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:10:13.012951 4292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:13.018890 4292 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38817
I20260926 01:10:13.018931 4420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38817 every 8 connection(s)
I20260926 01:10:13.019253 4292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384993574762-392-0/minicluster-data/ts-2/data/info.pb
I20260926 01:10:13.024276 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 4292
I20260926 01:10:13.024380 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384993574762-392-0/minicluster-data/ts-2/wal/instance
I20260926 01:10:13.024542 392 external_mini_cluster.cc:1428] Setting key b866497b121dd17e545820686106abc7
Sep 26 01:10:13 dist-test-slave-7b22 krb5kdc[3917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385012, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:10:13.028911 4002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:13.019941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49935 (local address 127.0.98.62:42373)
0926 01:10:13.020003 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:13.020005 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:13.020217 (+ 212us) server_negotiation.cc:407] Connection header received
0926 01:10:13.021175 (+ 958us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:13.021176 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:13.021190 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:13.021239 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:13.021903 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:13.022408 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:13.023465 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:13.023662 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:13.023719 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:13.026059 (+ 2340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:10:13.026060 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:10:13.026060 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:10:13.026067 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:10:13.027266 (+ 1199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:13.027640 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:13.027640 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:13.027641 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:13.027691 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:13.027995 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:13.027997 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:13.027998 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:13.028234 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:10:13.028357 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:13.028743 (+ 386us) server_negotiation.cc:299] Negotiation successful
0926 01:10:13.028835 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 01:10:13.028986 4423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:13.019932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42373 (local address 127.0.98.3:49935)
0926 01:10:13.020101 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:13.020139 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:13.021065 (+ 926us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:13.021324 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:13.021328 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:13.021594 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:10:13.021789 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:13.021793 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:13.022498 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:13.022499 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:13.023359 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:13.023361 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:13.023738 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:13.023739 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:13.023869 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:13.024370 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:10:13.024379 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:10:13.025952 (+ 1573us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:10:13.027343 (+ 1391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:10:13.027350 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:10:13.027352 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:10:13.027581 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:10:13.027741 (+ 160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:10:13.027742 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:10:13.027742 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:10:13.027848 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:10:13.028381 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:10:13.028387 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:10:13.028533 (+ 146us) client_negotiation.cc:769] Sending connection context
0926 01:10:13.028639 (+ 106us) client_negotiation.cc:240] Negotiation successful
0926 01:10:13.028783 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":88,"threads_started":1}
I20260926 01:10:13.029444 4421 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42373
I20260926 01:10:13.029548 4421 heartbeater.cc:464] Registering TS with master...
I20260926 01:10:13.029753 4421 heartbeater.cc:511] Master 127.0.98.62:42373 requested a full tablet report, sending...
I20260926 01:10:13.030198 3962 ts_manager.cc:194] Registered new tserver with Master: e3d486d1f720495b8e1e171a38cc03ad (127.0.98.3:38817)
I20260926 01:10:13.030573 3962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:49935
I20260926 01:10:13.036391 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 01:10:13 dist-test-slave-7b22 krb5kdc[3917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385013, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 01:10:13 dist-test-slave-7b22 krb5kdc[3917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385013, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 01:10:13.059656 4428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:13.051512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42986 (local address 127.0.98.1:36929)
0926 01:10:13.051652 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:13.051657 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:13.051670 (+ 13us) server_negotiation.cc:407] Connection header received
0926 01:10:13.051730 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:13.051733 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:13.051747 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:13.051825 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:13.052426 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:13.053188 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:13.054035 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:13.054362 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:13.054416 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:13.056788 (+ 2372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:10:13.056798 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:10:13.056798 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:10:13.056806 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:10:13.058449 (+ 1643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:13.058828 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:13.058832 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:13.058832 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:13.058918 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:13.059149 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:13.059149 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:13.059149 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:13.059295 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:10:13.059403 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:13.059483 (+ 80us) server_negotiation.cc:299] Negotiation successful
0926 01:10:13.059555 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":65,"threads_started":1}
Sep 26 01:10:13 dist-test-slave-7b22 krb5kdc[3917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385013, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:10:13.067987 4002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:13.061141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54682 (local address 127.0.98.62:42373)
0926 01:10:13.061197 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:13.061199 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:13.061209 (+ 10us) server_negotiation.cc:407] Connection header received
0926 01:10:13.061274 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:13.061275 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:13.061284 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:13.061322 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:13.061781 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:13.062322 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:13.063068 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:13.063267 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:13.063338 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:13.065522 (+ 2184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:10:13.065523 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:10:13.065524 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:10:13.065531 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:10:13.066847 (+ 1316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:13.067178 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:13.067180 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:13.067180 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:13.067235 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:13.067446 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:13.067446 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:13.067446 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:13.067587 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:10:13.067702 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:13.067843 (+ 141us) server_negotiation.cc:299] Negotiation successful
0926 01:10:13.067919 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
W20260926 01:10:13.068218 3962 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54682
I20260926 01:10:13.068540 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 4024
I20260926 01:10:13.072654 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 4158
I20260926 01:10:13.076385 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 4292
I20260926 01:10:13.080037 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 3930
2026-09-26T01:10:13Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1793 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384993574762-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 01:10:13 dist-test-slave-7b22 krb5kdc[4433](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 01:10:13 dist-test-slave-7b22 krb5kdc[4433](info): set up 2 sockets
Sep 26 01:10:13 dist-test-slave-7b22 krb5kdc[4433](info): commencing operation
krb5kdc: starting...
W20260926 01:10:14.041723 392 mini_kdc.cc:121] Time spent starting KDC: real 0.943s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384993574762-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384993574762-392-0/krb5kdc/test-user.keytab.
Sep 26 01:10:14 dist-test-slave-7b22 krb5kdc[4433](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385014, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T01:10:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:10:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384993574762-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384993574762-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 01:10:14.164175 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384993574762-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384993574762-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384993574762-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42815
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384993574762-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42815
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:10:14.246644 4446 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:10:14.246889 4446 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:10:14.246913 4446 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:10:14.248196 4446 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:10:14.248247 4446 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:10:14.248261 4446 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:10:14.248272 4446 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:10:14.248282 4446 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:10:14.250001 4446 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384993574762-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384993574762-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42815
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384993574762-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42815
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4446
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384993574762-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:10:14.250250 4446 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:10:14.250433 4446 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:10:14.255319 4454 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:10:14.255445 4446 server_base.cc:1034] running on GCE node
W20260926 01:10:14.255316 4452 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:10:14.255532 4451 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:10:14.255877 4446 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:10:14.256217 4446 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:10:14.257380 4446 hybrid_clock.cc:648] HybridClock initialized: now 1790385014257361 us; error 36 us; skew 500 ppm
Sep 26 01:10:14 dist-test-slave-7b22 krb5kdc[4433](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385014, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:10:14.260004 4446 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 01:10:14.260345 4446 fs_manager.cc:359] Metadata directory not provided
I20260926 01:10:14.260392 4446 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:10:14.260501 4446 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:10:14.261579 4446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384993574762-392-0/minicluster-data/master-0/data/instance:
uuid: "473fc41fa0d4431e8ef28ecd671e88dc"
format_stamp: "Formatted at 2026-09-26 01:10:14 on dist-test-slave-7b22"
server_key: "5a1e6d70bca2f236dc09f5828f63334a"
server_key_iv: "40b1a41d68199482084d29be26c9c60a"
server_key_version: "encryptionkey@0"
I20260926 01:10:14.261945 4446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384993574762-392-0/minicluster-data/master-0/wal/instance:
uuid: "473fc41fa0d4431e8ef28ecd671e88dc"
format_stamp: "Formatted at 2026-09-26 01:10:14 on dist-test-slave-7b22"
server_key: "5a1e6d70bca2f236dc09f5828f63334a"
server_key_iv: "40b1a41d68199482084d29be26c9c60a"
server_key_version: "encryptionkey@0"
I20260926 01:10:14.263187 4446 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 01:10:14.263895 4460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:14.264083 4446 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 01:10:14.264180 4446 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384993574762-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384993574762-392-0/minicluster-data/master-0/wal
uuid: "473fc41fa0d4431e8ef28ecd671e88dc"
format_stamp: "Formatted at 2026-09-26 01:10:14 on dist-test-slave-7b22"
server_key: "5a1e6d70bca2f236dc09f5828f63334a"
server_key_iv: "40b1a41d68199482084d29be26c9c60a"
server_key_version: "encryptionkey@0"
I20260926 01:10:14.264246 4446 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384993574762-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384993574762-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384993574762-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:10:14.279877 4446 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:10:14.282145 4446 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:10:14.282290 4446 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:10:14.285949 4446 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42815
I20260926 01:10:14.285976 4512 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42815 every 8 connection(s)
I20260926 01:10:14.286257 4446 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
I20260926 01:10:14.287202 4513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:10:14.289083 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 4446
I20260926 01:10:14.289191 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384993574762-392-0/minicluster-data/master-0/wal/instance
I20260926 01:10:14.289354 392 external_mini_cluster.cc:1428] Setting key 7034475a9688d81cf623dfa8a5491960
I20260926 01:10:14.289710 4513 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 473fc41fa0d4431e8ef28ecd671e88dc: Bootstrap starting.
I20260926 01:10:14.290453 4513 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 473fc41fa0d4431e8ef28ecd671e88dc: Neither blocks nor log segments found. Creating new log.
I20260926 01:10:14.290784 4513 log.cc:824] T 00000000000000000000000000000000 P 473fc41fa0d4431e8ef28ecd671e88dc: Log is configured to *not* fsync() on all Append() calls
I20260926 01:10:14.291546 4513 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 473fc41fa0d4431e8ef28ecd671e88dc: No bootstrap required, opened a new log
I20260926 01:10:14.293869 4513 raft_consensus.cc:348] T 00000000000000000000000000000000 P 473fc41fa0d4431e8ef28ecd671e88dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "473fc41fa0d4431e8ef28ecd671e88dc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42815 } }
I20260926 01:10:14.294056 4513 raft_consensus.cc:374] T 00000000000000000000000000000000 P 473fc41fa0d4431e8ef28ecd671e88dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:10:14.294092 4513 raft_consensus.cc:729] T 00000000000000000000000000000000 P 473fc41fa0d4431e8ef28ecd671e88dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 473fc41fa0d4431e8ef28ecd671e88dc, State: Initialized, Role: FOLLOWER
I20260926 01:10:14.294227 4513 consensus_queue.cc:260] T 00000000000000000000000000000000 P 473fc41fa0d4431e8ef28ecd671e88dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "473fc41fa0d4431e8ef28ecd671e88dc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42815 } }
I20260926 01:10:14.294301 4513 raft_consensus.cc:388] T 00000000000000000000000000000000 P 473fc41fa0d4431e8ef28ecd671e88dc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:10:14.294334 4513 raft_consensus.cc:482] T 00000000000000000000000000000000 P 473fc41fa0d4431e8ef28ecd671e88dc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:10:14.294378 4513 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 473fc41fa0d4431e8ef28ecd671e88dc [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:10:14.295084 4513 raft_consensus.cc:504] T 00000000000000000000000000000000 P 473fc41fa0d4431e8ef28ecd671e88dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "473fc41fa0d4431e8ef28ecd671e88dc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42815 } }
I20260926 01:10:14.295192 4513 leader_election.cc:302] T 00000000000000000000000000000000 P 473fc41fa0d4431e8ef28ecd671e88dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 473fc41fa0d4431e8ef28ecd671e88dc; no voters:
I20260926 01:10:14.295370 4513 leader_election.cc:288] T 00000000000000000000000000000000 P 473fc41fa0d4431e8ef28ecd671e88dc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:10:14.295403 4518 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 473fc41fa0d4431e8ef28ecd671e88dc [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:10:14.295547 4518 raft_consensus.cc:686] T 00000000000000000000000000000000 P 473fc41fa0d4431e8ef28ecd671e88dc [term 1 LEADER]: Becoming Leader. State: Replica: 473fc41fa0d4431e8ef28ecd671e88dc, State: Running, Role: LEADER
I20260926 01:10:14.295656 4513 sys_catalog.cc:564] T 00000000000000000000000000000000 P 473fc41fa0d4431e8ef28ecd671e88dc [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:10:14.295651 4518 consensus_queue.cc:237] T 00000000000000000000000000000000 P 473fc41fa0d4431e8ef28ecd671e88dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "473fc41fa0d4431e8ef28ecd671e88dc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42815 } }
I20260926 01:10:14.296149 4519 sys_catalog.cc:455] T 00000000000000000000000000000000 P 473fc41fa0d4431e8ef28ecd671e88dc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 473fc41fa0d4431e8ef28ecd671e88dc. Latest consensus state: current_term: 1 leader_uuid: "473fc41fa0d4431e8ef28ecd671e88dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "473fc41fa0d4431e8ef28ecd671e88dc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42815 } } }
I20260926 01:10:14.296232 4519 sys_catalog.cc:458] T 00000000000000000000000000000000 P 473fc41fa0d4431e8ef28ecd671e88dc [sys.catalog]: This master's current role is: LEADER
I20260926 01:10:14.296386 4520 sys_catalog.cc:455] T 00000000000000000000000000000000 P 473fc41fa0d4431e8ef28ecd671e88dc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "473fc41fa0d4431e8ef28ecd671e88dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "473fc41fa0d4431e8ef28ecd671e88dc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42815 } } }
I20260926 01:10:14.296446 4520 sys_catalog.cc:458] T 00000000000000000000000000000000 P 473fc41fa0d4431e8ef28ecd671e88dc [sys.catalog]: This master's current role is: LEADER
Sep 26 01:10:14 dist-test-slave-7b22 krb5kdc[4433](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385014, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:10:14.296872 4523 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:10:14.297389 4523 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:10:14.298982 4523 catalog_manager.cc:1347] Generated new cluster ID: 3faf70f8ee9149b285fabb2608298bab
I20260926 01:10:14.299032 4523 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 01:10:14.301008 4516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:14.289934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44154 (local address 127.0.98.62:42815)
0926 01:10:14.290133 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:14.290143 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:14.290181 (+ 38us) server_negotiation.cc:407] Connection header received
0926 01:10:14.290787 (+ 606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:14.290797 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:14.290866 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:14.291030 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:14.291694 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:14.293134 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:14.293998 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:14.294637 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:14.294755 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:14.297416 (+ 2661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:10:14.297421 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:10:14.297421 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:10:14.297469 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:10:14.299413 (+ 1944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:14.299821 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:14.299824 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:14.299824 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:14.299922 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:14.300112 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:14.300113 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:14.300113 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:14.300401 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:10:14.300533 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:14.300668 (+ 135us) server_negotiation.cc:299] Negotiation successful
0926 01:10:14.300797 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":92,"threads_started":1}
I20260926 01:10:14.308717 4523 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:10:14.309324 4523 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:10:14.315723 4523 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 473fc41fa0d4431e8ef28ecd671e88dc: Generated new TSK 0
I20260926 01:10:14.315938 4523 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384993574762-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384993574762-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 01:10:14.349279 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384993574762-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384993574762-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384993574762-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384993574762-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42815
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384993574762-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:10:14.443512 4539 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:10:14.443761 4539 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:10:14.443797 4539 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:10:14.445093 4539 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:10:14.445138 4539 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:10:14.445180 4539 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 01:10:14.446889 4539 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384993574762-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384993574762-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384993574762-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384993574762-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42815
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4539
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384993574762-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:10:14.447153 4539 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:10:14.447381 4539 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:10:14.452574 4544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:10:14.452615 4545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:10:14.452574 4547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:10:14.452625 4539 server_base.cc:1034] running on GCE node
I20260926 01:10:14.453043 4539 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:10:14.453294 4539 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:10:14.454444 4539 hybrid_clock.cc:648] HybridClock initialized: now 1790385014454431 us; error 32 us; skew 500 ppm
Sep 26 01:10:14 dist-test-slave-7b22 krb5kdc[4433](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385014, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:10:14.457229 4539 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 01:10:14.457587 4539 fs_manager.cc:359] Metadata directory not provided
I20260926 01:10:14.457640 4539 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:10:14.457813 4539 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:10:14.459007 4539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384993574762-392-0/minicluster-data/ts-0/data/instance:
uuid: "f5aaea0f13e943b4ac048fe3b7bf378b"
format_stamp: "Formatted at 2026-09-26 01:10:14 on dist-test-slave-7b22"
server_key: "297eb12f3854fce8f1d9189e5a53e372"
server_key_iv: "7921873cfaf12c49cfa5544057f62cef"
server_key_version: "encryptionkey@0"
I20260926 01:10:14.459348 4539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384993574762-392-0/minicluster-data/ts-0/wal/instance:
uuid: "f5aaea0f13e943b4ac048fe3b7bf378b"
format_stamp: "Formatted at 2026-09-26 01:10:14 on dist-test-slave-7b22"
server_key: "297eb12f3854fce8f1d9189e5a53e372"
server_key_iv: "7921873cfaf12c49cfa5544057f62cef"
server_key_version: "encryptionkey@0"
I20260926 01:10:14.460644 4539 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 01:10:14.461481 4553 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:14.461680 4539 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:10:14.461812 4539 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384993574762-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384993574762-392-0/minicluster-data/ts-0/wal
uuid: "f5aaea0f13e943b4ac048fe3b7bf378b"
format_stamp: "Formatted at 2026-09-26 01:10:14 on dist-test-slave-7b22"
server_key: "297eb12f3854fce8f1d9189e5a53e372"
server_key_iv: "7921873cfaf12c49cfa5544057f62cef"
server_key_version: "encryptionkey@0"
I20260926 01:10:14.462126 4539 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384993574762-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384993574762-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384993574762-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:10:14.482059 4539 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:10:14.484634 4539 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:10:14.484797 4539 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:10:14.485039 4539 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:10:14.485359 4539 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:10:14.485394 4539 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:14.485423 4539 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:10:14.485440 4539 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:14.491036 4539 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34263
I20260926 01:10:14.491091 4666 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34263 every 8 connection(s)
I20260926 01:10:14.491340 4539 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384993574762-392-0/minicluster-data/ts-0/data/info.pb
I20260926 01:10:14.495155 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 4539
I20260926 01:10:14.495275 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384993574762-392-0/minicluster-data/ts-0/wal/instance
I20260926 01:10:14.495463 392 external_mini_cluster.cc:1428] Setting key 03549b05127ed6c2dbf332b47079c958
Sep 26 01:10:14 dist-test-slave-7b22 krb5kdc[4433](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385014, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:10:14.502934 4516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:14.492323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48699 (local address 127.0.98.62:42815)
0926 01:10:14.492409 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:14.492412 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:14.492584 (+ 172us) server_negotiation.cc:407] Connection header received
0926 01:10:14.493122 (+ 538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:14.493123 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:14.493136 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:14.493190 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:14.494276 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:14.494790 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:14.496413 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:14.496631 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:14.496688 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:14.499100 (+ 2412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:10:14.499102 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:10:14.499102 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:10:14.499109 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:10:14.501102 (+ 1993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:14.501633 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:14.501634 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:14.501635 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:14.501704 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:14.502093 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:14.502095 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:14.502095 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:14.502255 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:10:14.502314 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:14.502683 (+ 369us) server_negotiation.cc:299] Negotiation successful
0926 01:10:14.502856 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 01:10:14.503247 4669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:14.492251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42815 (local address 127.0.98.1:48699)
0926 01:10:14.492445 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:14.492500 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:14.493018 (+ 518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:14.493295 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:14.493298 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:14.493598 (+ 300us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:10:14.494180 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:14.494187 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:14.494874 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:14.494875 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:14.496287 (+ 1412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:14.496290 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:14.496704 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:14.496705 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:14.496837 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:14.497461 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:10:14.497471 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:10:14.498985 (+ 1514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:10:14.501205 (+ 2220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:10:14.501214 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:10:14.501217 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:10:14.501547 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:10:14.501768 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:10:14.501769 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:10:14.501769 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:10:14.501925 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:10:14.502320 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:10:14.502326 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:10:14.502521 (+ 195us) client_negotiation.cc:769] Sending connection context
0926 01:10:14.502643 (+ 122us) client_negotiation.cc:240] Negotiation successful
0926 01:10:14.502843 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":94,"threads_started":1}
I20260926 01:10:14.503871 4667 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42815
I20260926 01:10:14.503985 4667 heartbeater.cc:464] Registering TS with master...
I20260926 01:10:14.504166 4667 heartbeater.cc:511] Master 127.0.98.62:42815 requested a full tablet report, sending...
I20260926 01:10:14.504726 4477 ts_manager.cc:194] Registered new tserver with Master: f5aaea0f13e943b4ac048fe3b7bf378b (127.0.98.1:34263)
I20260926 01:10:14.505656 4477 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:48699
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384993574762-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384993574762-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 01:10:14.524569 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384993574762-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384993574762-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384993574762-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384993574762-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42815
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384993574762-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:10:14.608804 4672 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:10:14.609036 4672 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:10:14.609061 4672 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:10:14.610304 4672 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:10:14.610349 4672 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:10:14.610376 4672 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 01:10:14.611881 4672 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384993574762-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384993574762-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384993574762-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384993574762-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42815
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4672
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384993574762-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:10:14.612080 4672 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:10:14.612264 4672 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:10:14.617643 4677 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:10:14.617745 4680 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:10:14.617638 4678 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:10:14.617992 4672 server_base.cc:1034] running on GCE node
I20260926 01:10:14.618310 4672 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:10:14.618726 4672 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:10:14.619915 4672 hybrid_clock.cc:648] HybridClock initialized: now 1790385014619885 us; error 42 us; skew 500 ppm
Sep 26 01:10:14 dist-test-slave-7b22 krb5kdc[4433](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385014, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:10:14.622546 4672 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 01:10:14.622912 4672 fs_manager.cc:359] Metadata directory not provided
I20260926 01:10:14.622972 4672 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:10:14.623092 4672 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:10:14.624179 4672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384993574762-392-0/minicluster-data/ts-1/data/instance:
uuid: "c93c82bd6103418ea98cc506ad6e336b"
format_stamp: "Formatted at 2026-09-26 01:10:14 on dist-test-slave-7b22"
server_key: "c0f3a12f8ff9a132cbf301060238a2da"
server_key_iv: "d778aeb362bf8ac9273b454c30d9e7f3"
server_key_version: "encryptionkey@0"
I20260926 01:10:14.624513 4672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384993574762-392-0/minicluster-data/ts-1/wal/instance:
uuid: "c93c82bd6103418ea98cc506ad6e336b"
format_stamp: "Formatted at 2026-09-26 01:10:14 on dist-test-slave-7b22"
server_key: "c0f3a12f8ff9a132cbf301060238a2da"
server_key_iv: "d778aeb362bf8ac9273b454c30d9e7f3"
server_key_version: "encryptionkey@0"
I20260926 01:10:14.625775 4672 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 01:10:14.626564 4686 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:14.626740 4672 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:10:14.626816 4672 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384993574762-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384993574762-392-0/minicluster-data/ts-1/wal
uuid: "c93c82bd6103418ea98cc506ad6e336b"
format_stamp: "Formatted at 2026-09-26 01:10:14 on dist-test-slave-7b22"
server_key: "c0f3a12f8ff9a132cbf301060238a2da"
server_key_iv: "d778aeb362bf8ac9273b454c30d9e7f3"
server_key_version: "encryptionkey@0"
I20260926 01:10:14.626885 4672 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384993574762-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384993574762-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384993574762-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:10:14.654788 4672 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:10:14.656875 4672 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:10:14.657011 4672 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:10:14.657238 4672 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:10:14.657539 4672 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:10:14.657570 4672 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:14.657600 4672 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:10:14.657624 4672 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:14.663194 4672 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33855
I20260926 01:10:14.663232 4799 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33855 every 8 connection(s)
I20260926 01:10:14.663489 4672 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384993574762-392-0/minicluster-data/ts-1/data/info.pb
Sep 26 01:10:14 dist-test-slave-7b22 krb5kdc[4433](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385014, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:10:14.670532 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 4672
I20260926 01:10:14.670634 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384993574762-392-0/minicluster-data/ts-1/wal/instance
I20260926 01:10:14.670806 392 external_mini_cluster.cc:1428] Setting key ead98b05a5d38b18e1d92b2c281288f0
I20260926 01:10:14.673267 4516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:14.664197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47291 (local address 127.0.98.62:42815)
0926 01:10:14.664258 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:14.664260 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:14.664492 (+ 232us) server_negotiation.cc:407] Connection header received
0926 01:10:14.665004 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:14.665006 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:14.665016 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:14.665067 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:14.666106 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:14.666614 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:14.667760 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:14.667955 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:14.668018 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:14.670110 (+ 2092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:10:14.670111 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:10:14.670112 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:10:14.670119 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:10:14.671675 (+ 1556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:14.672205 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:14.672206 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:14.672206 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:14.672263 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:14.672542 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:14.672543 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:14.672543 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:14.672673 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:10:14.672747 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:14.673069 (+ 322us) server_negotiation.cc:299] Negotiation successful
0926 01:10:14.673158 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 01:10:14.673434 4802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:14.664198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42815 (local address 127.0.98.2:47291)
0926 01:10:14.664373 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:14.664412 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:14.664904 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:14.665129 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:14.665132 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:14.665392 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:10:14.666019 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:14.666026 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:14.666748 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:14.666748 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:14.667690 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:14.667693 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:14.668025 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:14.668026 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:14.668133 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:14.668626 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:10:14.668634 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:10:14.670019 (+ 1385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:10:14.671769 (+ 1750us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:10:14.671777 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:10:14.671779 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:10:14.672106 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:10:14.672322 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:10:14.672322 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:10:14.672323 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:10:14.672454 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:10:14.672737 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:10:14.672742 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:10:14.672926 (+ 184us) client_negotiation.cc:769] Sending connection context
0926 01:10:14.673041 (+ 115us) client_negotiation.cc:240] Negotiation successful
0926 01:10:14.673206 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":92,"threads_started":1}
I20260926 01:10:14.673947 4800 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42815
I20260926 01:10:14.674038 4800 heartbeater.cc:464] Registering TS with master...
I20260926 01:10:14.674189 4800 heartbeater.cc:511] Master 127.0.98.62:42815 requested a full tablet report, sending...
I20260926 01:10:14.674592 4477 ts_manager.cc:194] Registered new tserver with Master: c93c82bd6103418ea98cc506ad6e336b (127.0.98.2:33855)
I20260926 01:10:14.674960 4477 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:47291
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384993574762-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384993574762-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 01:10:14.696699 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384993574762-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384993574762-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384993574762-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384993574762-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42815
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384993574762-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:10:14.779771 4805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:10:14.780007 4805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:10:14.780041 4805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:10:14.781303 4805 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:10:14.781345 4805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:10:14.781383 4805 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 01:10:14.783068 4805 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384993574762-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384993574762-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384993574762-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384993574762-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42815
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4805
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384993574762-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:10:14.783329 4805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:10:14.783543 4805 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:10:14.788586 4811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:10:14.788594 4813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:10:14.788884 4805 server_base.cc:1034] running on GCE node
W20260926 01:10:14.788594 4810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:10:14.789139 4805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:10:14.789402 4805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:10:14.790566 4805 hybrid_clock.cc:648] HybridClock initialized: now 1790385014790545 us; error 33 us; skew 500 ppm
Sep 26 01:10:14 dist-test-slave-7b22 krb5kdc[4433](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385014, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:10:14.793367 4805 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 01:10:14.793747 4805 fs_manager.cc:359] Metadata directory not provided
I20260926 01:10:14.793802 4805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:10:14.793951 4805 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:10:14.795029 4805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384993574762-392-0/minicluster-data/ts-2/data/instance:
uuid: "2f3ca20cac7346159db3f33d0fd587fc"
format_stamp: "Formatted at 2026-09-26 01:10:14 on dist-test-slave-7b22"
server_key: "8f41611526060966f532e921f47c08a4"
server_key_iv: "625041f5bbf7937775c68d15b93bc20b"
server_key_version: "encryptionkey@0"
I20260926 01:10:14.795333 4805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384993574762-392-0/minicluster-data/ts-2/wal/instance:
uuid: "2f3ca20cac7346159db3f33d0fd587fc"
format_stamp: "Formatted at 2026-09-26 01:10:14 on dist-test-slave-7b22"
server_key: "8f41611526060966f532e921f47c08a4"
server_key_iv: "625041f5bbf7937775c68d15b93bc20b"
server_key_version: "encryptionkey@0"
I20260926 01:10:14.796748 4805 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 01:10:14.797504 4819 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:14.797644 4805 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:10:14.797772 4805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384993574762-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384993574762-392-0/minicluster-data/ts-2/wal
uuid: "2f3ca20cac7346159db3f33d0fd587fc"
format_stamp: "Formatted at 2026-09-26 01:10:14 on dist-test-slave-7b22"
server_key: "8f41611526060966f532e921f47c08a4"
server_key_iv: "625041f5bbf7937775c68d15b93bc20b"
server_key_version: "encryptionkey@0"
I20260926 01:10:14.797878 4805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384993574762-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384993574762-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384993574762-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:10:14.814253 4805 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:10:14.816558 4805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:10:14.816676 4805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:10:14.816881 4805 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:10:14.817145 4805 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:10:14.817174 4805 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:14.817196 4805 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:10:14.817207 4805 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:14.822778 4805 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45173
I20260926 01:10:14.822842 4932 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45173 every 8 connection(s)
I20260926 01:10:14.823084 4805 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384993574762-392-0/minicluster-data/ts-2/data/info.pb
Sep 26 01:10:14 dist-test-slave-7b22 krb5kdc[4433](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385014, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:10:14.831959 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 4805
I20260926 01:10:14.832072 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384993574762-392-0/minicluster-data/ts-2/wal/instance
I20260926 01:10:14.832254 392 external_mini_cluster.cc:1428] Setting key a56b4b3f0c2c234cdf18c30bde56228e
I20260926 01:10:14.833032 4516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:14.823845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44457 (local address 127.0.98.62:42815)
0926 01:10:14.823942 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:14.823948 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:14.824021 (+ 73us) server_negotiation.cc:407] Connection header received
0926 01:10:14.824534 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:14.824536 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:14.824546 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:14.824614 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:14.825550 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:14.826129 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:14.827145 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:14.827341 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:14.827394 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:14.829569 (+ 2175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:10:14.829570 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:10:14.829570 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:10:14.829579 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:10:14.830981 (+ 1402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:14.831414 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:14.831415 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:14.831415 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:14.831478 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:14.832055 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:14.832056 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:14.832057 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:14.832230 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:10:14.832319 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:14.832844 (+ 525us) server_negotiation.cc:299] Negotiation successful
0926 01:10:14.832934 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260926 01:10:14.833232 4935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:14.823758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42815 (local address 127.0.98.3:44457)
0926 01:10:14.823908 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:14.823951 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:14.824421 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:14.824668 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:14.824672 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:14.824942 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:10:14.825438 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:14.825444 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:14.826233 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:14.826235 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:14.827068 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:14.827071 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:14.827421 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:14.827421 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:14.827529 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:14.827999 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:10:14.828007 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:10:14.829473 (+ 1466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:10:14.831074 (+ 1601us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:10:14.831081 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:10:14.831083 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:10:14.831338 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:10:14.831683 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:10:14.831685 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:10:14.831685 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:10:14.831906 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:10:14.832363 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:10:14.832372 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:10:14.832564 (+ 192us) client_negotiation.cc:769] Sending connection context
0926 01:10:14.832771 (+ 207us) client_negotiation.cc:240] Negotiation successful
0926 01:10:14.832964 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":63,"threads_started":1}
I20260926 01:10:14.833952 4933 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42815
I20260926 01:10:14.834102 4933 heartbeater.cc:464] Registering TS with master...
I20260926 01:10:14.834308 4933 heartbeater.cc:511] Master 127.0.98.62:42815 requested a full tablet report, sending...
I20260926 01:10:14.834916 4477 ts_manager.cc:194] Registered new tserver with Master: 2f3ca20cac7346159db3f33d0fd587fc (127.0.98.3:45173)
I20260926 01:10:14.835311 4477 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:44457
I20260926 01:10:14.844638 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 01:10:14.852763 4516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:14.846171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44162 (local address 127.0.98.62:42815)
0926 01:10:14.846239 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:14.846241 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:14.846292 (+ 51us) server_negotiation.cc:407] Connection header received
0926 01:10:14.846321 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:14.846322 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:14.846342 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:14.846391 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:14.847007 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:14.847532 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:14.848280 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:14.848490 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:14.848548 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:14.849945 (+ 1397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:10:14.849947 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:10:14.849947 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:10:14.849955 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:10:14.851368 (+ 1413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:14.851760 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:14.851762 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:14.851762 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:14.851820 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:14.852107 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:14.852108 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:14.852108 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:14.852295 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:10:14.852394 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:14.852531 (+ 137us) server_negotiation.cc:299] Negotiation successful
0926 01:10:14.852616 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 01:10:14.854729 4477 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44162:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:10:14.856357 4477 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:10:14.863062 4946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:14.858827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45173 (local address 127.0.0.1:42842)
0926 01:10:14.859054 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:14.859064 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:14.859118 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:14.859750 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:14.859750 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:14.859752 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:10:14.859854 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:14.859856 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:14.861074 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:14.861075 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:14.862037 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:14.862039 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:14.862733 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:14.862734 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:14.862836 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:14.862852 (+ 16us) client_negotiation.cc:769] Sending connection context
0926 01:10:14.862915 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 01:10:14.862971 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"mutex_wait_us":3,"thread_start_us":95,"threads_started":1}
I20260926 01:10:14.863390 4945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:14.858806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33855 (local address 127.0.0.1:40616)
0926 01:10:14.858994 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:14.859039 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:14.859249 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:14.859724 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:14.859725 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:14.859727 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:10:14.859829 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:14.859831 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:14.860803 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:14.860805 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:14.862151 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:14.862176 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:14.863063 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:14.863064 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:14.863121 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:14.863226 (+ 105us) client_negotiation.cc:769] Sending connection context
0926 01:10:14.863273 (+ 47us) client_negotiation.cc:240] Negotiation successful
0926 01:10:14.863315 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"mutex_wait_us":43,"thread_start_us":51,"threads_started":1}
I20260926 01:10:14.863416 4949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:14.859274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42842 (local address 127.0.98.3:45173)
0926 01:10:14.859587 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:14.859592 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:14.859601 (+ 9us) server_negotiation.cc:407] Connection header received
0926 01:10:14.859642 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:14.859644 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:14.859659 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:14.859925 (+ 266us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:10:14.859991 (+ 66us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:14.860996 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:14.862108 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:14.862653 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:14.862721 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:14.862784 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:14.863283 (+ 499us) server_negotiation.cc:299] Negotiation successful
0926 01:10:14.863330 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":102,"threads_started":1}
I20260926 01:10:14.863636 4947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:14.859081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40616 (local address 127.0.98.2:33855)
0926 01:10:14.859487 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:14.859491 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:14.859501 (+ 10us) server_negotiation.cc:407] Connection header received
0926 01:10:14.859536 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:14.859538 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:14.859552 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:14.859797 (+ 245us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:10:14.859877 (+ 80us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:14.860676 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:14.862287 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:14.862997 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:14.863053 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:14.863107 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:14.863504 (+ 397us) server_negotiation.cc:299] Negotiation successful
0926 01:10:14.863557 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":91,"threads_started":1}
I20260926 01:10:14.864291 4944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:14.858809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34263 (local address 127.0.0.1:60030)
0926 01:10:14.858994 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:14.859195 (+ 201us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:14.859226 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:14.859481 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:14.859484 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:14.859486 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:10:14.859564 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:14.859566 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:14.860909 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:14.860910 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:14.862844 (+ 1934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:14.862846 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:14.863811 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:14.863812 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:14.863884 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:14.863887 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 01:10:14.864091 (+ 204us) client_negotiation.cc:240] Negotiation successful
0926 01:10:14.864191 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"mutex_wait_us":847,"thread_start_us":107,"threads_started":1}
I20260926 01:10:14.864234 4734 tablet_service.cc:1467] Processing CreateTablet for tablet 7d483aa6eb114d5a9802497c45c81f5c (DEFAULT_TABLE table=test-table [id=c2b51ec93f8c4c468737805069bf2511]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:10:14.864225 4867 tablet_service.cc:1467] Processing CreateTablet for tablet 7d483aa6eb114d5a9802497c45c81f5c (DEFAULT_TABLE table=test-table [id=c2b51ec93f8c4c468737805069bf2511]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:10:14.864576 4867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d483aa6eb114d5a9802497c45c81f5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:10:14.864576 4734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d483aa6eb114d5a9802497c45c81f5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:10:14.864713 4948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:14.859088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60030 (local address 127.0.98.1:34263)
0926 01:10:14.859277 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:14.859282 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:14.859294 (+ 12us) server_negotiation.cc:407] Connection header received
0926 01:10:14.859340 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:14.859342 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:14.859359 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:14.859460 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:10:14.859649 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:14.860775 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:14.862930 (+ 2155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:14.863669 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:14.863726 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:14.863780 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:14.864530 (+ 750us) server_negotiation.cc:299] Negotiation successful
0926 01:10:14.864608 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":84,"threads_started":1}
I20260926 01:10:14.865581 4601 tablet_service.cc:1467] Processing CreateTablet for tablet 7d483aa6eb114d5a9802497c45c81f5c (DEFAULT_TABLE table=test-table [id=c2b51ec93f8c4c468737805069bf2511]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:10:14.866052 4601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d483aa6eb114d5a9802497c45c81f5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:10:14.868136 4950 tablet_bootstrap.cc:492] T 7d483aa6eb114d5a9802497c45c81f5c P 2f3ca20cac7346159db3f33d0fd587fc: Bootstrap starting.
I20260926 01:10:14.868364 4951 tablet_bootstrap.cc:492] T 7d483aa6eb114d5a9802497c45c81f5c P c93c82bd6103418ea98cc506ad6e336b: Bootstrap starting.
I20260926 01:10:14.868927 4951 tablet_bootstrap.cc:654] T 7d483aa6eb114d5a9802497c45c81f5c P c93c82bd6103418ea98cc506ad6e336b: Neither blocks nor log segments found. Creating new log.
I20260926 01:10:14.869117 4952 tablet_bootstrap.cc:492] T 7d483aa6eb114d5a9802497c45c81f5c P f5aaea0f13e943b4ac048fe3b7bf378b: Bootstrap starting.
I20260926 01:10:14.868957 4950 tablet_bootstrap.cc:654] T 7d483aa6eb114d5a9802497c45c81f5c P 2f3ca20cac7346159db3f33d0fd587fc: Neither blocks nor log segments found. Creating new log.
I20260926 01:10:14.869539 4951 log.cc:824] T 7d483aa6eb114d5a9802497c45c81f5c P c93c82bd6103418ea98cc506ad6e336b: Log is configured to *not* fsync() on all Append() calls
I20260926 01:10:14.869640 4950 log.cc:824] T 7d483aa6eb114d5a9802497c45c81f5c P 2f3ca20cac7346159db3f33d0fd587fc: Log is configured to *not* fsync() on all Append() calls
I20260926 01:10:14.869717 4952 tablet_bootstrap.cc:654] T 7d483aa6eb114d5a9802497c45c81f5c P f5aaea0f13e943b4ac048fe3b7bf378b: Neither blocks nor log segments found. Creating new log.
I20260926 01:10:14.870275 4952 log.cc:824] T 7d483aa6eb114d5a9802497c45c81f5c P f5aaea0f13e943b4ac048fe3b7bf378b: Log is configured to *not* fsync() on all Append() calls
I20260926 01:10:14.870359 4951 tablet_bootstrap.cc:492] T 7d483aa6eb114d5a9802497c45c81f5c P c93c82bd6103418ea98cc506ad6e336b: No bootstrap required, opened a new log
I20260926 01:10:14.870368 4950 tablet_bootstrap.cc:492] T 7d483aa6eb114d5a9802497c45c81f5c P 2f3ca20cac7346159db3f33d0fd587fc: No bootstrap required, opened a new log
I20260926 01:10:14.870432 4950 ts_tablet_manager.cc:1397] T 7d483aa6eb114d5a9802497c45c81f5c P 2f3ca20cac7346159db3f33d0fd587fc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260926 01:10:14.870432 4951 ts_tablet_manager.cc:1397] T 7d483aa6eb114d5a9802497c45c81f5c P c93c82bd6103418ea98cc506ad6e336b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 01:10:14.870877 4952 tablet_bootstrap.cc:492] T 7d483aa6eb114d5a9802497c45c81f5c P f5aaea0f13e943b4ac048fe3b7bf378b: No bootstrap required, opened a new log
I20260926 01:10:14.870934 4952 ts_tablet_manager.cc:1397] T 7d483aa6eb114d5a9802497c45c81f5c P f5aaea0f13e943b4ac048fe3b7bf378b: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 01:10:14.872257 4952 raft_consensus.cc:348] T 7d483aa6eb114d5a9802497c45c81f5c P f5aaea0f13e943b4ac048fe3b7bf378b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5aaea0f13e943b4ac048fe3b7bf378b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34263 } } peers { permanent_uuid: "2f3ca20cac7346159db3f33d0fd587fc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45173 } } peers { permanent_uuid: "c93c82bd6103418ea98cc506ad6e336b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33855 } }
I20260926 01:10:14.872402 4952 raft_consensus.cc:374] T 7d483aa6eb114d5a9802497c45c81f5c P f5aaea0f13e943b4ac048fe3b7bf378b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:10:14.872434 4952 raft_consensus.cc:729] T 7d483aa6eb114d5a9802497c45c81f5c P f5aaea0f13e943b4ac048fe3b7bf378b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5aaea0f13e943b4ac048fe3b7bf378b, State: Initialized, Role: FOLLOWER
I20260926 01:10:14.872414 4951 raft_consensus.cc:348] T 7d483aa6eb114d5a9802497c45c81f5c P c93c82bd6103418ea98cc506ad6e336b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5aaea0f13e943b4ac048fe3b7bf378b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34263 } } peers { permanent_uuid: "2f3ca20cac7346159db3f33d0fd587fc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45173 } } peers { permanent_uuid: "c93c82bd6103418ea98cc506ad6e336b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33855 } }
I20260926 01:10:14.872414 4950 raft_consensus.cc:348] T 7d483aa6eb114d5a9802497c45c81f5c P 2f3ca20cac7346159db3f33d0fd587fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5aaea0f13e943b4ac048fe3b7bf378b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34263 } } peers { permanent_uuid: "2f3ca20cac7346159db3f33d0fd587fc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45173 } } peers { permanent_uuid: "c93c82bd6103418ea98cc506ad6e336b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33855 } }
I20260926 01:10:14.872574 4951 raft_consensus.cc:374] T 7d483aa6eb114d5a9802497c45c81f5c P c93c82bd6103418ea98cc506ad6e336b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:10:14.872574 4950 raft_consensus.cc:374] T 7d483aa6eb114d5a9802497c45c81f5c P 2f3ca20cac7346159db3f33d0fd587fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:10:14.872555 4952 consensus_queue.cc:260] T 7d483aa6eb114d5a9802497c45c81f5c P f5aaea0f13e943b4ac048fe3b7bf378b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5aaea0f13e943b4ac048fe3b7bf378b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34263 } } peers { permanent_uuid: "2f3ca20cac7346159db3f33d0fd587fc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45173 } } peers { permanent_uuid: "c93c82bd6103418ea98cc506ad6e336b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33855 } }
I20260926 01:10:14.872612 4950 raft_consensus.cc:729] T 7d483aa6eb114d5a9802497c45c81f5c P 2f3ca20cac7346159db3f33d0fd587fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f3ca20cac7346159db3f33d0fd587fc, State: Initialized, Role: FOLLOWER
I20260926 01:10:14.872612 4951 raft_consensus.cc:729] T 7d483aa6eb114d5a9802497c45c81f5c P c93c82bd6103418ea98cc506ad6e336b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c93c82bd6103418ea98cc506ad6e336b, State: Initialized, Role: FOLLOWER
I20260926 01:10:14.872717 4951 consensus_queue.cc:260] T 7d483aa6eb114d5a9802497c45c81f5c P c93c82bd6103418ea98cc506ad6e336b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5aaea0f13e943b4ac048fe3b7bf378b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34263 } } peers { permanent_uuid: "2f3ca20cac7346159db3f33d0fd587fc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45173 } } peers { permanent_uuid: "c93c82bd6103418ea98cc506ad6e336b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33855 } }
I20260926 01:10:14.872751 4950 consensus_queue.cc:260] T 7d483aa6eb114d5a9802497c45c81f5c P 2f3ca20cac7346159db3f33d0fd587fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5aaea0f13e943b4ac048fe3b7bf378b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34263 } } peers { permanent_uuid: "2f3ca20cac7346159db3f33d0fd587fc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45173 } } peers { permanent_uuid: "c93c82bd6103418ea98cc506ad6e336b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33855 } }
I20260926 01:10:14.872817 4952 ts_tablet_manager.cc:1428] T 7d483aa6eb114d5a9802497c45c81f5c P f5aaea0f13e943b4ac048fe3b7bf378b: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260926 01:10:14.872947 4667 heartbeater.cc:503] Master 127.0.98.62:42815 was elected leader, sending a full tablet report...
I20260926 01:10:14.873014 4951 ts_tablet_manager.cc:1428] T 7d483aa6eb114d5a9802497c45c81f5c P c93c82bd6103418ea98cc506ad6e336b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260926 01:10:14.873210 4800 heartbeater.cc:503] Master 127.0.98.62:42815 was elected leader, sending a full tablet report...
I20260926 01:10:14.873528 4950 ts_tablet_manager.cc:1428] T 7d483aa6eb114d5a9802497c45c81f5c P 2f3ca20cac7346159db3f33d0fd587fc: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260926 01:10:14.873664 4933 heartbeater.cc:503] Master 127.0.98.62:42815 was elected leader, sending a full tablet report...
I20260926 01:10:14.907577 4956 raft_consensus.cc:482] T 7d483aa6eb114d5a9802497c45c81f5c P f5aaea0f13e943b4ac048fe3b7bf378b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:10:14.907790 4956 raft_consensus.cc:504] T 7d483aa6eb114d5a9802497c45c81f5c P f5aaea0f13e943b4ac048fe3b7bf378b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5aaea0f13e943b4ac048fe3b7bf378b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34263 } } peers { permanent_uuid: "2f3ca20cac7346159db3f33d0fd587fc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45173 } } peers { permanent_uuid: "c93c82bd6103418ea98cc506ad6e336b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33855 } }
I20260926 01:10:14.908145 4956 leader_election.cc:288] T 7d483aa6eb114d5a9802497c45c81f5c P f5aaea0f13e943b4ac048fe3b7bf378b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2f3ca20cac7346159db3f33d0fd587fc (127.0.98.3:45173), c93c82bd6103418ea98cc506ad6e336b (127.0.98.2:33855)
I20260926 01:10:14.911216 4949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:14.908581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48503 (local address 127.0.98.3:45173)
0926 01:10:14.908738 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:14.908740 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:14.908755 (+ 15us) server_negotiation.cc:407] Connection header received
0926 01:10:14.908880 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:14.908881 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:14.908892 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:14.908943 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:10:14.909093 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:14.909543 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:14.910548 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:14.910915 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:14.910976 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:14.910987 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:14.911111 (+ 124us) server_negotiation.cc:299] Negotiation successful
0926 01:10:14.911148 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 01:10:14.911233 4959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:14.908400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45173 (local address 127.0.98.1:48503)
0926 01:10:14.908712 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:14.908720 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:14.908764 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:14.908959 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:14.908960 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:14.908963 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:10:14.909031 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:14.909032 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:14.909609 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:14.909610 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:14.910452 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:14.910455 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:14.910984 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:14.910984 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:14.911049 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:14.911053 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 01:10:14.911098 (+ 45us) client_negotiation.cc:240] Negotiation successful
0926 01:10:14.911138 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"mutex_wait_us":143,"thread_start_us":109,"threads_started":1}
I20260926 01:10:14.911409 4669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:14.908400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33855 (local address 127.0.98.1:44353)
0926 01:10:14.908442 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:14.908454 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:14.908497 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:14.908875 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:14.908876 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:14.908878 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:10:14.908923 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:14.908925 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:14.909546 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:14.909547 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:14.910555 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:14.910558 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:14.911164 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:14.911172 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:14.911237 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:14.911240 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 01:10:14.911286 (+ 46us) client_negotiation.cc:240] Negotiation successful
0926 01:10:14.911328 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22,"mutex_wait_us":102}
I20260926 01:10:14.911407 4947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:14.908616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44353 (local address 127.0.98.2:33855)
0926 01:10:14.908716 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:14.908719 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:14.908747 (+ 28us) server_negotiation.cc:407] Connection header received
0926 01:10:14.908776 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:14.908777 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:14.908790 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:14.908855 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:10:14.908995 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:14.909462 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:14.910719 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:14.911089 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:14.911158 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:14.911168 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:14.911302 (+ 134us) server_negotiation.cc:299] Negotiation successful
0926 01:10:14.911341 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 01:10:14.911609 4887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d483aa6eb114d5a9802497c45c81f5c" candidate_uuid: "f5aaea0f13e943b4ac048fe3b7bf378b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f3ca20cac7346159db3f33d0fd587fc" is_pre_election: true
I20260926 01:10:14.911734 4754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d483aa6eb114d5a9802497c45c81f5c" candidate_uuid: "f5aaea0f13e943b4ac048fe3b7bf378b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c93c82bd6103418ea98cc506ad6e336b" is_pre_election: true
I20260926 01:10:14.911790 4887 raft_consensus.cc:2454] T 7d483aa6eb114d5a9802497c45c81f5c P 2f3ca20cac7346159db3f33d0fd587fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5aaea0f13e943b4ac048fe3b7bf378b in term 0.
I20260926 01:10:14.911881 4754 raft_consensus.cc:2454] T 7d483aa6eb114d5a9802497c45c81f5c P c93c82bd6103418ea98cc506ad6e336b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5aaea0f13e943b4ac048fe3b7bf378b in term 0.
I20260926 01:10:14.912127 4554 leader_election.cc:302] T 7d483aa6eb114d5a9802497c45c81f5c P f5aaea0f13e943b4ac048fe3b7bf378b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2f3ca20cac7346159db3f33d0fd587fc, c93c82bd6103418ea98cc506ad6e336b, f5aaea0f13e943b4ac048fe3b7bf378b; no voters:
I20260926 01:10:14.912292 4956 raft_consensus.cc:2781] T 7d483aa6eb114d5a9802497c45c81f5c P f5aaea0f13e943b4ac048fe3b7bf378b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:10:14.912352 4956 raft_consensus.cc:482] T 7d483aa6eb114d5a9802497c45c81f5c P f5aaea0f13e943b4ac048fe3b7bf378b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:10:14.912369 4956 raft_consensus.cc:3037] T 7d483aa6eb114d5a9802497c45c81f5c P f5aaea0f13e943b4ac048fe3b7bf378b [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:10:14.913041 4956 raft_consensus.cc:504] T 7d483aa6eb114d5a9802497c45c81f5c P f5aaea0f13e943b4ac048fe3b7bf378b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5aaea0f13e943b4ac048fe3b7bf378b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34263 } } peers { permanent_uuid: "2f3ca20cac7346159db3f33d0fd587fc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45173 } } peers { permanent_uuid: "c93c82bd6103418ea98cc506ad6e336b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33855 } }
I20260926 01:10:14.913172 4956 leader_election.cc:288] T 7d483aa6eb114d5a9802497c45c81f5c P f5aaea0f13e943b4ac048fe3b7bf378b [CANDIDATE]: Term 1 election: Requested vote from peers 2f3ca20cac7346159db3f33d0fd587fc (127.0.98.3:45173), c93c82bd6103418ea98cc506ad6e336b (127.0.98.2:33855)
I20260926 01:10:14.913326 4754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d483aa6eb114d5a9802497c45c81f5c" candidate_uuid: "f5aaea0f13e943b4ac048fe3b7bf378b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c93c82bd6103418ea98cc506ad6e336b"
I20260926 01:10:14.913385 4887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d483aa6eb114d5a9802497c45c81f5c" candidate_uuid: "f5aaea0f13e943b4ac048fe3b7bf378b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f3ca20cac7346159db3f33d0fd587fc"
I20260926 01:10:14.913403 4754 raft_consensus.cc:3037] T 7d483aa6eb114d5a9802497c45c81f5c P c93c82bd6103418ea98cc506ad6e336b [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:10:14.913460 4887 raft_consensus.cc:3037] T 7d483aa6eb114d5a9802497c45c81f5c P 2f3ca20cac7346159db3f33d0fd587fc [term 0 FOLLOWER]: Advancing to term 1
W20260926 01:10:14.913949 4801 tablet.cc:2369] T 7d483aa6eb114d5a9802497c45c81f5c P c93c82bd6103418ea98cc506ad6e336b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 01:10:14.914242 4887 raft_consensus.cc:2454] T 7d483aa6eb114d5a9802497c45c81f5c P 2f3ca20cac7346159db3f33d0fd587fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5aaea0f13e943b4ac048fe3b7bf378b in term 1.
I20260926 01:10:14.914250 4754 raft_consensus.cc:2454] T 7d483aa6eb114d5a9802497c45c81f5c P c93c82bd6103418ea98cc506ad6e336b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5aaea0f13e943b4ac048fe3b7bf378b in term 1.
I20260926 01:10:14.914414 4557 leader_election.cc:302] T 7d483aa6eb114d5a9802497c45c81f5c P f5aaea0f13e943b4ac048fe3b7bf378b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c93c82bd6103418ea98cc506ad6e336b, f5aaea0f13e943b4ac048fe3b7bf378b; no voters:
I20260926 01:10:14.914497 4956 raft_consensus.cc:2781] T 7d483aa6eb114d5a9802497c45c81f5c P f5aaea0f13e943b4ac048fe3b7bf378b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:10:14.914666 4956 raft_consensus.cc:686] T 7d483aa6eb114d5a9802497c45c81f5c P f5aaea0f13e943b4ac048fe3b7bf378b [term 1 LEADER]: Becoming Leader. State: Replica: f5aaea0f13e943b4ac048fe3b7bf378b, State: Running, Role: LEADER
I20260926 01:10:14.914749 4956 consensus_queue.cc:237] T 7d483aa6eb114d5a9802497c45c81f5c P f5aaea0f13e943b4ac048fe3b7bf378b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5aaea0f13e943b4ac048fe3b7bf378b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34263 } } peers { permanent_uuid: "2f3ca20cac7346159db3f33d0fd587fc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45173 } } peers { permanent_uuid: "c93c82bd6103418ea98cc506ad6e336b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33855 } }
I20260926 01:10:14.915715 4476 catalog_manager.cc:5495] T 7d483aa6eb114d5a9802497c45c81f5c P f5aaea0f13e943b4ac048fe3b7bf378b reported cstate change: term changed from 0 to 1, leader changed from <none> to f5aaea0f13e943b4ac048fe3b7bf378b (127.0.98.1). New cstate: current_term: 1 leader_uuid: "f5aaea0f13e943b4ac048fe3b7bf378b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5aaea0f13e943b4ac048fe3b7bf378b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f3ca20cac7346159db3f33d0fd587fc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c93c82bd6103418ea98cc506ad6e336b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33855 } health_report { overall_health: UNKNOWN } } }
I20260926 01:10:14.925486 4948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:14.923150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60036 (local address 127.0.98.1:34263)
0926 01:10:14.923201 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:14.923202 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:14.923211 (+ 9us) server_negotiation.cc:407] Connection header received
0926 01:10:14.923260 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:14.923261 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:14.923280 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:14.923339 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:10:14.923428 (+ 89us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:14.923893 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:14.924725 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:14.924947 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:14.925014 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:14.925166 (+ 152us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:10:14.925270 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:10:14.925330 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:14.925370 (+ 40us) server_negotiation.cc:299] Negotiation successful
0926 01:10:14.925390 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260926 01:10:14.927642 4887 raft_consensus.cc:1261] T 7d483aa6eb114d5a9802497c45c81f5c P 2f3ca20cac7346159db3f33d0fd587fc [term 1 FOLLOWER]: Refusing update from remote peer f5aaea0f13e943b4ac048fe3b7bf378b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:10:14.927642 4754 raft_consensus.cc:1261] T 7d483aa6eb114d5a9802497c45c81f5c P c93c82bd6103418ea98cc506ad6e336b [term 1 FOLLOWER]: Refusing update from remote peer f5aaea0f13e943b4ac048fe3b7bf378b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:10:14.928013 4956 consensus_queue.cc:1035] T 7d483aa6eb114d5a9802497c45c81f5c P f5aaea0f13e943b4ac048fe3b7bf378b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c93c82bd6103418ea98cc506ad6e336b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:10:14.928153 4960 consensus_queue.cc:1035] T 7d483aa6eb114d5a9802497c45c81f5c P f5aaea0f13e943b4ac048fe3b7bf378b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f3ca20cac7346159db3f33d0fd587fc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:10:14.930883 4964 mvcc.cc:205] Tried to move back new op lower bound from 7333417021139914752 to 7333417021091676160. Current Snapshot: MvccSnapshot[applied={T|T < 7333417021139914752}]
I20260926 01:10:14.933610 4477 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44162:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:10:14.933715 4477 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44162:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:10:14.934432 4477 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 473fc41fa0d4431e8ef28ecd671e88dc: Sending DeleteTablet for 3 replicas of tablet 7d483aa6eb114d5a9802497c45c81f5c
I20260926 01:10:14.934762 4601 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d483aa6eb114d5a9802497c45c81f5c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:10:14 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60030
I20260926 01:10:14.934780 4867 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d483aa6eb114d5a9802497c45c81f5c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:10:14 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42842
I20260926 01:10:14.934823 4734 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d483aa6eb114d5a9802497c45c81f5c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:10:14 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40616
I20260926 01:10:14.934993 4974 tablet_replica.cc:330] T 7d483aa6eb114d5a9802497c45c81f5c P 2f3ca20cac7346159db3f33d0fd587fc: stopping tablet replica
I20260926 01:10:14.935093 4974 raft_consensus.cc:2229] T 7d483aa6eb114d5a9802497c45c81f5c P 2f3ca20cac7346159db3f33d0fd587fc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:10:14.935194 4974 raft_consensus.cc:2258] T 7d483aa6eb114d5a9802497c45c81f5c P 2f3ca20cac7346159db3f33d0fd587fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:10:14.935250 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 4539
I20260926 01:10:14.935631 4974 ts_tablet_manager.cc:1905] T 7d483aa6eb114d5a9802497c45c81f5c P 2f3ca20cac7346159db3f33d0fd587fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:10:14.935914 4975 tablet_replica.cc:330] T 7d483aa6eb114d5a9802497c45c81f5c P c93c82bd6103418ea98cc506ad6e336b: stopping tablet replica
I20260926 01:10:14.936028 4975 raft_consensus.cc:2229] T 7d483aa6eb114d5a9802497c45c81f5c P c93c82bd6103418ea98cc506ad6e336b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:10:14.936136 4975 raft_consensus.cc:2258] T 7d483aa6eb114d5a9802497c45c81f5c P c93c82bd6103418ea98cc506ad6e336b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:10:14.937383 4974 ts_tablet_manager.cc:1918] T 7d483aa6eb114d5a9802497c45c81f5c P 2f3ca20cac7346159db3f33d0fd587fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:10:14.937457 4974 log.cc:1196] T 7d483aa6eb114d5a9802497c45c81f5c P 2f3ca20cac7346159db3f33d0fd587fc: Deleting WAL directory at /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384993574762-392-0/minicluster-data/ts-2/wal/wals/7d483aa6eb114d5a9802497c45c81f5c
I20260926 01:10:14.937728 4974 ts_tablet_manager.cc:1939] T 7d483aa6eb114d5a9802497c45c81f5c P 2f3ca20cac7346159db3f33d0fd587fc: Deleting consensus metadata
I20260926 01:10:14.938151 4461 catalog_manager.cc:4841] TS 2f3ca20cac7346159db3f33d0fd587fc (127.0.98.3:45173): tablet 7d483aa6eb114d5a9802497c45c81f5c (table test-table [id=c2b51ec93f8c4c468737805069bf2511]) successfully deleted
W20260926 01:10:14.939939 4462 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34263 (error 108)
I20260926 01:10:14.940289 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 4672
I20260926 01:10:14.940481 4975 ts_tablet_manager.cc:1905] T 7d483aa6eb114d5a9802497c45c81f5c P c93c82bd6103418ea98cc506ad6e336b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:10:14.940922 4944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:14.940352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34263 (local address 127.0.0.1:60042)
0926 01:10:14.940848 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:14.940891 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":473}
W20260926 01:10:14.941478 4462 catalog_manager.cc:4584] TS f5aaea0f13e943b4ac048fe3b7bf378b (127.0.98.1:34263): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7d483aa6eb114d5a9802497c45c81f5c: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34263: connect: Connection refused (error 111)
I20260926 01:10:14.944485 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 4805
I20260926 01:10:14.944535 4944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:14.944420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33855 (local address 127.0.0.1:40618)
0926 01:10:14.944472 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:14.944502 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:33855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260926 01:10:14.948735 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 4446
2026-09-26T01:10:14Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (1869 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-26T01:10:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:10:14Z Disabled control of system clock
I20260926 01:10:14.975411 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384993574762-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384993574762-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384993574762-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45611
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:40027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45611
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 01:10:15.058969 4980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:10:15.059209 4980 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:10:15.059242 4980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:10:15.060618 4980 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:10:15.060670 4980 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:10:15.060683 4980 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:10:15.060698 4980 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:10:15.060709 4980 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:10:15.062167 4980 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384993574762-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384993574762-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45611
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45611
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4980
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384993574762-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:10:15.062402 4980 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:10:15.062669 4980 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:10:15.067480 4988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:10:15.067591 4980 server_base.cc:1034] running on GCE node
W20260926 01:10:15.067483 4985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:10:15.067480 4986 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:10:15.067992 4980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:10:15.068320 4980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:10:15.069484 4980 hybrid_clock.cc:648] HybridClock initialized: now 1790385015069468 us; error 32 us; skew 500 ppm
I20260926 01:10:15.070753 4980 webserver.cc:460] Webserver started at http://127.0.98.62:44723/ using document root <none> and password file <none>
I20260926 01:10:15.070986 4980 fs_manager.cc:359] Metadata directory not provided
I20260926 01:10:15.071033 4980 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:10:15.071166 4980 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:10:15.072304 4980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384993574762-392-0/minicluster-data/master-0/data/instance:
uuid: "3b3da3a1543b47d8997e19c61ff531fe"
format_stamp: "Formatted at 2026-09-26 01:10:15 on dist-test-slave-7b22"
server_key: "69ed6b9c6a229c8058fa81befaab1348"
server_key_iv: "3055ca3b4eb0dfd6aa8a9391138248cd"
server_key_version: "encryptionkey@0"
I20260926 01:10:15.072635 4980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384993574762-392-0/minicluster-data/master-0/wal/instance:
uuid: "3b3da3a1543b47d8997e19c61ff531fe"
format_stamp: "Formatted at 2026-09-26 01:10:15 on dist-test-slave-7b22"
server_key: "69ed6b9c6a229c8058fa81befaab1348"
server_key_iv: "3055ca3b4eb0dfd6aa8a9391138248cd"
server_key_version: "encryptionkey@0"
I20260926 01:10:15.074249 4980 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 01:10:15.075424 4994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:15.075719 4980 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:10:15.075829 4980 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384993574762-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384993574762-392-0/minicluster-data/master-0/wal
uuid: "3b3da3a1543b47d8997e19c61ff531fe"
format_stamp: "Formatted at 2026-09-26 01:10:15 on dist-test-slave-7b22"
server_key: "69ed6b9c6a229c8058fa81befaab1348"
server_key_iv: "3055ca3b4eb0dfd6aa8a9391138248cd"
server_key_version: "encryptionkey@0"
I20260926 01:10:15.075904 4980 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384993574762-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384993574762-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384993574762-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:10:15.079553 4980 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:10:15.079872 4980 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:10:15.079996 4980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:10:15.083943 4980 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45611
I20260926 01:10:15.084128 5046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45611 every 8 connection(s)
I20260926 01:10:15.084357 4980 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
I20260926 01:10:15.085192 5047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:10:15.088177 5047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b3da3a1543b47d8997e19c61ff531fe: Bootstrap starting.
I20260926 01:10:15.088785 5047 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3b3da3a1543b47d8997e19c61ff531fe: Neither blocks nor log segments found. Creating new log.
I20260926 01:10:15.089068 5047 log.cc:824] T 00000000000000000000000000000000 P 3b3da3a1543b47d8997e19c61ff531fe: Log is configured to *not* fsync() on all Append() calls
I20260926 01:10:15.089767 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 4980
I20260926 01:10:15.089967 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384993574762-392-0/minicluster-data/master-0/wal/instance
I20260926 01:10:15.090124 392 external_mini_cluster.cc:1428] Setting key 43c741b64008b6aa72d0ab94d0813962
I20260926 01:10:15.090260 5047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b3da3a1543b47d8997e19c61ff531fe: No bootstrap required, opened a new log
I20260926 01:10:15.092392 5047 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3b3da3a1543b47d8997e19c61ff531fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b3da3a1543b47d8997e19c61ff531fe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45611 } }
I20260926 01:10:15.092583 5047 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3b3da3a1543b47d8997e19c61ff531fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:10:15.092624 5047 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3b3da3a1543b47d8997e19c61ff531fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b3da3a1543b47d8997e19c61ff531fe, State: Initialized, Role: FOLLOWER
I20260926 01:10:15.092764 5047 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b3da3a1543b47d8997e19c61ff531fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b3da3a1543b47d8997e19c61ff531fe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45611 } }
I20260926 01:10:15.092854 5047 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3b3da3a1543b47d8997e19c61ff531fe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:10:15.092886 5047 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3b3da3a1543b47d8997e19c61ff531fe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:10:15.092934 5047 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3b3da3a1543b47d8997e19c61ff531fe [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:10:15.093569 5051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:15.090631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47424 (local address 127.0.98.62:45611)
0926 01:10:15.090837 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:15.090845 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:15.090870 (+ 25us) server_negotiation.cc:407] Connection header received
0926 01:10:15.091531 (+ 661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:15.091538 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:15.091582 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:15.091711 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:15.092897 (+ 1186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:10:15.092903 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:10:15.092903 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:10:15.092911 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:10:15.092997 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:10:15.093016 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:10:15.093093 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:15.093182 (+ 89us) server_negotiation.cc:299] Negotiation successful
0926 01:10:15.093262 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":89,"threads_started":1}
I20260926 01:10:15.093578 5047 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3b3da3a1543b47d8997e19c61ff531fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b3da3a1543b47d8997e19c61ff531fe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45611 } }
I20260926 01:10:15.093713 5047 leader_election.cc:302] T 00000000000000000000000000000000 P 3b3da3a1543b47d8997e19c61ff531fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3b3da3a1543b47d8997e19c61ff531fe; no voters:
I20260926 01:10:15.093905 5047 leader_election.cc:288] T 00000000000000000000000000000000 P 3b3da3a1543b47d8997e19c61ff531fe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:10:15.094168 5052 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3b3da3a1543b47d8997e19c61ff531fe [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:10:15.094321 5052 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3b3da3a1543b47d8997e19c61ff531fe [term 1 LEADER]: Becoming Leader. State: Replica: 3b3da3a1543b47d8997e19c61ff531fe, State: Running, Role: LEADER
I20260926 01:10:15.094414 5052 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3b3da3a1543b47d8997e19c61ff531fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b3da3a1543b47d8997e19c61ff531fe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45611 } }
I20260926 01:10:15.094437 5047 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3b3da3a1543b47d8997e19c61ff531fe [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:10:15.095624 5053 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b3da3a1543b47d8997e19c61ff531fe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b3da3a1543b47d8997e19c61ff531fe. Latest consensus state: current_term: 1 leader_uuid: "3b3da3a1543b47d8997e19c61ff531fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b3da3a1543b47d8997e19c61ff531fe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45611 } } }
I20260926 01:10:15.095722 5053 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b3da3a1543b47d8997e19c61ff531fe [sys.catalog]: This master's current role is: LEADER
I20260926 01:10:15.095788 5054 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b3da3a1543b47d8997e19c61ff531fe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3b3da3a1543b47d8997e19c61ff531fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b3da3a1543b47d8997e19c61ff531fe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45611 } } }
I20260926 01:10:15.095971 5054 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b3da3a1543b47d8997e19c61ff531fe [sys.catalog]: This master's current role is: LEADER
W20260926 01:10:15.096182 5067 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3b3da3a1543b47d8997e19c61ff531fe: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 01:10:15.096246 5067 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 01:10:15.096374 5069 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:10:15.096653 5069 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:10:15.098428 5069 catalog_manager.cc:1347] Generated new cluster ID: db242ab98df94f67bf96f38216c8b6af
I20260926 01:10:15.098483 5069 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 01:10:15.118893 5069 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:10:15.119107 5069 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:10:15.124668 5069 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3b3da3a1543b47d8997e19c61ff531fe: Generated new TSK 0
I20260926 01:10:15.124867 5069 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 01:10:15.133033 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384993574762-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384993574762-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384993574762-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384993574762-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45611
--builtin_ntp_servers=127.0.98.20:40027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 01:10:15.225627 5071 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:10:15.225869 5071 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:10:15.225904 5071 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:10:15.227228 5071 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:10:15.227275 5071 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:10:15.227313 5071 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 01:10:15.228847 5071 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384993574762-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384993574762-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384993574762-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45611
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5071
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384993574762-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:10:15.229113 5071 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:10:15.229411 5071 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:10:15.234599 5076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:10:15.234599 5079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:10:15.234607 5077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:10:15.235033 5071 server_base.cc:1034] running on GCE node
I20260926 01:10:15.235253 5071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:10:15.235488 5071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:10:15.236670 5071 hybrid_clock.cc:648] HybridClock initialized: now 1790385015236652 us; error 30 us; skew 500 ppm
I20260926 01:10:15.238078 5071 webserver.cc:460] Webserver started at http://127.0.98.1:43359/ using document root <none> and password file <none>
I20260926 01:10:15.238346 5071 fs_manager.cc:359] Metadata directory not provided
I20260926 01:10:15.238401 5071 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:10:15.238525 5071 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:10:15.239638 5071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384993574762-392-0/minicluster-data/ts-0/data/instance:
uuid: "078c9dbc14de40e289b6aab1df4e64b9"
format_stamp: "Formatted at 2026-09-26 01:10:15 on dist-test-slave-7b22"
server_key: "7e251ded3552a073daf40ec16f635a41"
server_key_iv: "b9dbac5d8115470750eef0b8fc7222b5"
server_key_version: "encryptionkey@0"
I20260926 01:10:15.239986 5071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384993574762-392-0/minicluster-data/ts-0/wal/instance:
uuid: "078c9dbc14de40e289b6aab1df4e64b9"
format_stamp: "Formatted at 2026-09-26 01:10:15 on dist-test-slave-7b22"
server_key: "7e251ded3552a073daf40ec16f635a41"
server_key_iv: "b9dbac5d8115470750eef0b8fc7222b5"
server_key_version: "encryptionkey@0"
I20260926 01:10:15.242050 5071 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260926 01:10:15.243487 5085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:15.243887 5071 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:10:15.244005 5071 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384993574762-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384993574762-392-0/minicluster-data/ts-0/wal
uuid: "078c9dbc14de40e289b6aab1df4e64b9"
format_stamp: "Formatted at 2026-09-26 01:10:15 on dist-test-slave-7b22"
server_key: "7e251ded3552a073daf40ec16f635a41"
server_key_iv: "b9dbac5d8115470750eef0b8fc7222b5"
server_key_version: "encryptionkey@0"
I20260926 01:10:15.244099 5071 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384993574762-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384993574762-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384993574762-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:10:15.247790 5071 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:10:15.248088 5071 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:10:15.248214 5071 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:10:15.248446 5071 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:10:15.248759 5071 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:10:15.248802 5071 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:15.248835 5071 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:10:15.248859 5071 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:15.254980 5071 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43849
I20260926 01:10:15.255036 5198 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43849 every 8 connection(s)
I20260926 01:10:15.255445 5071 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384993574762-392-0/minicluster-data/ts-0/data/info.pb
I20260926 01:10:15.258605 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 5071
I20260926 01:10:15.258739 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384993574762-392-0/minicluster-data/ts-0/wal/instance
I20260926 01:10:15.258896 392 external_mini_cluster.cc:1428] Setting key 540f37c71f788a59f0de24eb4549706b
I20260926 01:10:15.259318 5051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:15.256549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38787 (local address 127.0.98.62:45611)
0926 01:10:15.256636 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:15.256638 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:15.256944 (+ 306us) server_negotiation.cc:407] Connection header received
0926 01:10:15.257465 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:15.257467 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:15.257472 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:15.257515 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:15.259045 (+ 1530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:10:15.259047 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:10:15.259047 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:10:15.259053 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:10:15.259071 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:10:15.259075 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:10:15.259132 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:15.259246 (+ 114us) server_negotiation.cc:299] Negotiation successful
0926 01:10:15.259263 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 01:10:15.259533 5201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:15.256430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45611 (local address 127.0.98.1:38787)
0926 01:10:15.256798 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:15.256837 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:15.257369 (+ 532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:15.257588 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:15.257592 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:15.257602 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:10:15.258877 (+ 1275us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 01:10:15.258878 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:10:15.258910 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 01:10:15.258951 (+ 41us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:10:15.259149 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:10:15.259156 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:10:15.259160 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 01:10:15.259253 (+ 93us) client_negotiation.cc:240] Negotiation successful
0926 01:10:15.259323 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":103,"threads_started":1}
I20260926 01:10:15.259949 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384993574762-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384993574762-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384993574762-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384993574762-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45611
--builtin_ntp_servers=127.0.98.20:40027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260926 01:10:15.260160 5199 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45611
I20260926 01:10:15.260251 5199 heartbeater.cc:464] Registering TS with master...
I20260926 01:10:15.260407 5199 heartbeater.cc:511] Master 127.0.98.62:45611 requested a full tablet report, sending...
I20260926 01:10:15.260950 5011 ts_manager.cc:194] Registered new tserver with Master: 078c9dbc14de40e289b6aab1df4e64b9 (127.0.98.1:43849)
W20260926 01:10:15.343662 5202 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:10:15.343907 5202 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:10:15.343940 5202 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:10:15.345222 5202 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:10:15.345268 5202 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:10:15.345305 5202 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 01:10:15.346778 5202 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384993574762-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384993574762-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384993574762-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45611
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5202
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384993574762-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:10:15.347158 5202 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:10:15.347501 5202 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:10:15.352782 5210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:10:15.352713 5208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:10:15.352801 5207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:10:15.352922 5202 server_base.cc:1034] running on GCE node
I20260926 01:10:15.353113 5202 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:10:15.353332 5202 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:10:15.354470 5202 hybrid_clock.cc:648] HybridClock initialized: now 1790385015354453 us; error 32 us; skew 500 ppm
I20260926 01:10:15.355773 5202 webserver.cc:460] Webserver started at http://127.0.98.2:42911/ using document root <none> and password file <none>
I20260926 01:10:15.355998 5202 fs_manager.cc:359] Metadata directory not provided
I20260926 01:10:15.356048 5202 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:10:15.356174 5202 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:10:15.357271 5202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384993574762-392-0/minicluster-data/ts-1/data/instance:
uuid: "ad2afcd7dff742ec8656f3faa395051e"
format_stamp: "Formatted at 2026-09-26 01:10:15 on dist-test-slave-7b22"
server_key: "270acd81ef4859a351c59c4e7dbcea0c"
server_key_iv: "f20222ab62dd16b69634c69b458fc896"
server_key_version: "encryptionkey@0"
I20260926 01:10:15.357610 5202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384993574762-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ad2afcd7dff742ec8656f3faa395051e"
format_stamp: "Formatted at 2026-09-26 01:10:15 on dist-test-slave-7b22"
server_key: "270acd81ef4859a351c59c4e7dbcea0c"
server_key_iv: "f20222ab62dd16b69634c69b458fc896"
server_key_version: "encryptionkey@0"
I20260926 01:10:15.358999 5202 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 01:10:15.359913 5216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:15.360145 5202 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:10:15.360241 5202 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384993574762-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384993574762-392-0/minicluster-data/ts-1/wal
uuid: "ad2afcd7dff742ec8656f3faa395051e"
format_stamp: "Formatted at 2026-09-26 01:10:15 on dist-test-slave-7b22"
server_key: "270acd81ef4859a351c59c4e7dbcea0c"
server_key_iv: "f20222ab62dd16b69634c69b458fc896"
server_key_version: "encryptionkey@0"
I20260926 01:10:15.360322 5202 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384993574762-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384993574762-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384993574762-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:10:15.363863 5202 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:10:15.364145 5202 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:10:15.364266 5202 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:10:15.364473 5202 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:10:15.364763 5202 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:10:15.364794 5202 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:15.364822 5202 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:10:15.364848 5202 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:15.370637 5202 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45773
I20260926 01:10:15.370678 5329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45773 every 8 connection(s)
I20260926 01:10:15.371053 5202 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384993574762-392-0/minicluster-data/ts-1/data/info.pb
I20260926 01:10:15.374470 5051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:15.371805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55753 (local address 127.0.98.62:45611)
0926 01:10:15.371867 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:15.371871 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:15.372158 (+ 287us) server_negotiation.cc:407] Connection header received
0926 01:10:15.372650 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:15.372652 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:15.372660 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:15.372720 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:15.374206 (+ 1486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:10:15.374207 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:10:15.374207 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:10:15.374212 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:10:15.374227 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:10:15.374231 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:10:15.374276 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:15.374393 (+ 117us) server_negotiation.cc:299] Negotiation successful
0926 01:10:15.374406 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 01:10:15.374672 5332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:15.371805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45611 (local address 127.0.98.2:55753)
0926 01:10:15.372038 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:15.372074 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:15.372545 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:15.372855 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:15.372860 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:15.372885 (+ 25us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:10:15.373969 (+ 1084us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 01:10:15.373969 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:10:15.374002 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 01:10:15.374034 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:10:15.374309 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:10:15.374315 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:10:15.374318 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 01:10:15.374397 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 01:10:15.374459 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":92,"threads_started":1}
I20260926 01:10:15.375116 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 5202
I20260926 01:10:15.375229 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384993574762-392-0/minicluster-data/ts-1/wal/instance
I20260926 01:10:15.375370 5330 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45611
I20260926 01:10:15.375417 392 external_mini_cluster.cc:1428] Setting key 0d20e7abc56273897befb6645796c026
I20260926 01:10:15.375465 5330 heartbeater.cc:464] Registering TS with master...
I20260926 01:10:15.375614 5330 heartbeater.cc:511] Master 127.0.98.62:45611 requested a full tablet report, sending...
I20260926 01:10:15.376078 5011 ts_manager.cc:194] Registered new tserver with Master: ad2afcd7dff742ec8656f3faa395051e (127.0.98.2:45773)
I20260926 01:10:15.376360 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384993574762-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384993574762-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384993574762-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384993574762-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45611
--builtin_ntp_servers=127.0.98.20:40027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 01:10:15.459425 5333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:10:15.459682 5333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:10:15.459707 5333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:10:15.460966 5333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:10:15.461014 5333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:10:15.461043 5333 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 01:10:15.462613 5333 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384993574762-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384993574762-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384993574762-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45611
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5333
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384993574762-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:10:15.462913 5333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:10:15.463248 5333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:10:15.468052 5341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:10:15.468072 5339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:10:15.468060 5338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:10:15.468343 5333 server_base.cc:1034] running on GCE node
I20260926 01:10:15.468572 5333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:10:15.468814 5333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:10:15.469986 5333 hybrid_clock.cc:648] HybridClock initialized: now 1790385015469977 us; error 36 us; skew 500 ppm
I20260926 01:10:15.471398 5333 webserver.cc:460] Webserver started at http://127.0.98.3:44237/ using document root <none> and password file <none>
I20260926 01:10:15.471637 5333 fs_manager.cc:359] Metadata directory not provided
I20260926 01:10:15.471685 5333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:10:15.471813 5333 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:10:15.472886 5333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384993574762-392-0/minicluster-data/ts-2/data/instance:
uuid: "d8f719bbbc2c4c4aad29dd33860292c6"
format_stamp: "Formatted at 2026-09-26 01:10:15 on dist-test-slave-7b22"
server_key: "497478d699e1939ebc2fd10f5e5291a1"
server_key_iv: "5c4d75ddb8624fdf0acfce66f4cb40cd"
server_key_version: "encryptionkey@0"
I20260926 01:10:15.473222 5333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384993574762-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d8f719bbbc2c4c4aad29dd33860292c6"
format_stamp: "Formatted at 2026-09-26 01:10:15 on dist-test-slave-7b22"
server_key: "497478d699e1939ebc2fd10f5e5291a1"
server_key_iv: "5c4d75ddb8624fdf0acfce66f4cb40cd"
server_key_version: "encryptionkey@0"
I20260926 01:10:15.474552 5333 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 01:10:15.475224 5347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:15.475363 5333 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 01:10:15.475445 5333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384993574762-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384993574762-392-0/minicluster-data/ts-2/wal
uuid: "d8f719bbbc2c4c4aad29dd33860292c6"
format_stamp: "Formatted at 2026-09-26 01:10:15 on dist-test-slave-7b22"
server_key: "497478d699e1939ebc2fd10f5e5291a1"
server_key_iv: "5c4d75ddb8624fdf0acfce66f4cb40cd"
server_key_version: "encryptionkey@0"
I20260926 01:10:15.475507 5333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384993574762-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384993574762-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384993574762-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:10:15.478935 5333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:10:15.479210 5333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:10:15.479334 5333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:10:15.479523 5333 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:10:15.479808 5333 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:10:15.479840 5333 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:15.479866 5333 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:10:15.479883 5333 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:15.485217 5333 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34841
I20260926 01:10:15.485281 5460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34841 every 8 connection(s)
I20260926 01:10:15.485575 5333 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384993574762-392-0/minicluster-data/ts-2/data/info.pb
I20260926 01:10:15.488605 5051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:15.486320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40893 (local address 127.0.98.62:45611)
0926 01:10:15.486376 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:15.486379 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:15.486590 (+ 211us) server_negotiation.cc:407] Connection header received
0926 01:10:15.487095 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:15.487097 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:15.487103 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:15.487160 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:15.488320 (+ 1160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:10:15.488321 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:10:15.488322 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:10:15.488353 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:10:15.488386 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:10:15.488390 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:10:15.488437 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:15.488512 (+ 75us) server_negotiation.cc:299] Negotiation successful
0926 01:10:15.488530 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 01:10:15.488803 5463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:15.486286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45611 (local address 127.0.98.3:40893)
0926 01:10:15.486473 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:15.486508 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:15.486993 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:15.487236 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:15.487240 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:15.487250 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:10:15.488161 (+ 911us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 01:10:15.488161 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:10:15.488190 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 01:10:15.488223 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:10:15.488443 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:10:15.488449 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:10:15.488452 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 01:10:15.488526 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 01:10:15.488591 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":79,"threads_started":1}
I20260926 01:10:15.489351 5461 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45611
I20260926 01:10:15.489431 5461 heartbeater.cc:464] Registering TS with master...
I20260926 01:10:15.489573 5461 heartbeater.cc:511] Master 127.0.98.62:45611 requested a full tablet report, sending...
I20260926 01:10:15.489944 5011 ts_manager.cc:194] Registered new tserver with Master: d8f719bbbc2c4c4aad29dd33860292c6 (127.0.98.3:34841)
I20260926 01:10:15.491482 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 5333
I20260926 01:10:15.491573 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384993574762-392-0/minicluster-data/ts-2/wal/instance
I20260926 01:10:15.491797 392 external_mini_cluster.cc:1428] Setting key 635e52fcb3cbb9b49605fb257478bb8b
I20260926 01:10:15.492717 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 01:10:15.495275 5051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:15.494077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47426 (local address 127.0.98.62:45611)
0926 01:10:15.494152 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:15.494154 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:15.494233 (+ 79us) server_negotiation.cc:407] Connection header received
0926 01:10:15.494283 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:15.494284 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:15.494298 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:15.494393 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:15.495069 (+ 676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:10:15.495071 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:10:15.495071 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:10:15.495076 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:10:15.495091 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:10:15.495095 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:10:15.495137 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:15.495186 (+ 49us) server_negotiation.cc:299] Negotiation successful
0926 01:10:15.495199 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 01:10:15.497140 5011 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47426:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:10:15.498874 5011 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:10:15.502911 5473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:15.501276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60420 (local address 127.0.98.1:43849)
0926 01:10:15.501477 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:15.501482 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:15.501493 (+ 11us) server_negotiation.cc:407] Connection header received
0926 01:10:15.501518 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:15.501520 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:15.501530 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:15.501592 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:15.502632 (+ 1040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:10:15.502639 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:10:15.502639 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:10:15.502643 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:10:15.502677 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:10:15.502698 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:10:15.502749 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:15.502812 (+ 63us) server_negotiation.cc:299] Negotiation successful
0926 01:10:15.502827 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":99,"threads_started":1}
I20260926 01:10:15.502947 5470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:15.501042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43849 (local address 127.0.0.1:60420)
0926 01:10:15.501196 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:15.501237 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:15.501294 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:15.501609 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:15.501614 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:15.501619 (+ 5us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:10:15.502158 (+ 539us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 01:10:15.502158 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:10:15.502537 (+ 379us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 01:10:15.502550 (+ 13us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:10:15.502746 (+ 196us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:10:15.502747 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:10:15.502748 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 01:10:15.502794 (+ 46us) client_negotiation.cc:240] Negotiation successful
0926 01:10:15.502808 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":90,"threads_started":1}
I20260926 01:10:15.502947 5469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:15.501054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45773 (local address 127.0.0.1:59512)
0926 01:10:15.501393 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:15.501405 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:15.501454 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:15.502094 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:15.502095 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:15.502097 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:10:15.502558 (+ 461us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 01:10:15.502559 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:10:15.502566 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 01:10:15.502577 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:10:15.502782 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:10:15.502783 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:10:15.502783 (+ 0us) client_negotiation.cc:769] Sending connection context
0926 01:10:15.502818 (+ 35us) client_negotiation.cc:240] Negotiation successful
0926 01:10:15.502832 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":100,"threads_started":1}
I20260926 01:10:15.502867 5471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:15.501042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34841 (local address 127.0.0.1:35472)
0926 01:10:15.501378 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:15.501385 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:15.501416 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:15.502049 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:15.502050 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:15.502053 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:10:15.502399 (+ 346us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 01:10:15.502400 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:10:15.502413 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 01:10:15.502440 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:10:15.502666 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:10:15.502672 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:10:15.502675 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 01:10:15.502744 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 01:10:15.502758 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"mutex_wait_us":3,"thread_start_us":56,"threads_started":1}
I20260926 01:10:15.503199 5472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:15.501218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35472 (local address 127.0.98.3:34841)
0926 01:10:15.501906 (+ 688us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:15.501910 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:15.501920 (+ 10us) server_negotiation.cc:407] Connection header received
0926 01:10:15.501946 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:15.501948 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:15.501957 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:15.502050 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:15.502530 (+ 480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:10:15.502538 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:10:15.502538 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:10:15.502543 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:10:15.502584 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:10:15.502615 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:10:15.502671 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:15.502907 (+ 236us) server_negotiation.cc:299] Negotiation successful
0926 01:10:15.503119 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":647,"thread_start_us":73,"threads_started":1}
I20260926 01:10:15.503634 5474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:15.501311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59512 (local address 127.0.98.2:45773)
0926 01:10:15.501949 (+ 638us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:15.501952 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:15.501961 (+ 9us) server_negotiation.cc:407] Connection header received
0926 01:10:15.501985 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:15.501988 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:15.501995 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:15.502245 (+ 250us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:15.502652 (+ 407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:10:15.502659 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:10:15.502659 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:10:15.502663 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:10:15.502696 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:10:15.502716 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:10:15.502909 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:15.502933 (+ 24us) server_negotiation.cc:299] Negotiation successful
0926 01:10:15.502946 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":81,"threads_started":1}
I20260926 01:10:15.503727 5133 tablet_service.cc:1467] Processing CreateTablet for tablet 5b87baddc1db4dafb1c6ba8a4859c37e (DEFAULT_TABLE table=test-table [id=4f6321bad73a486596d8a76191fb21d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:10:15.504098 5133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b87baddc1db4dafb1c6ba8a4859c37e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:10:15.504066 5395 tablet_service.cc:1467] Processing CreateTablet for tablet 5b87baddc1db4dafb1c6ba8a4859c37e (DEFAULT_TABLE table=test-table [id=4f6321bad73a486596d8a76191fb21d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:10:15.504271 5264 tablet_service.cc:1467] Processing CreateTablet for tablet 5b87baddc1db4dafb1c6ba8a4859c37e (DEFAULT_TABLE table=test-table [id=4f6321bad73a486596d8a76191fb21d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:10:15.504380 5395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b87baddc1db4dafb1c6ba8a4859c37e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:10:15.504549 5264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b87baddc1db4dafb1c6ba8a4859c37e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:10:15.507056 5475 tablet_bootstrap.cc:492] T 5b87baddc1db4dafb1c6ba8a4859c37e P 078c9dbc14de40e289b6aab1df4e64b9: Bootstrap starting.
I20260926 01:10:15.507670 5475 tablet_bootstrap.cc:654] T 5b87baddc1db4dafb1c6ba8a4859c37e P 078c9dbc14de40e289b6aab1df4e64b9: Neither blocks nor log segments found. Creating new log.
I20260926 01:10:15.507675 5476 tablet_bootstrap.cc:492] T 5b87baddc1db4dafb1c6ba8a4859c37e P d8f719bbbc2c4c4aad29dd33860292c6: Bootstrap starting.
I20260926 01:10:15.508164 5475 log.cc:824] T 5b87baddc1db4dafb1c6ba8a4859c37e P 078c9dbc14de40e289b6aab1df4e64b9: Log is configured to *not* fsync() on all Append() calls
I20260926 01:10:15.508469 5476 tablet_bootstrap.cc:654] T 5b87baddc1db4dafb1c6ba8a4859c37e P d8f719bbbc2c4c4aad29dd33860292c6: Neither blocks nor log segments found. Creating new log.
I20260926 01:10:15.508536 5477 tablet_bootstrap.cc:492] T 5b87baddc1db4dafb1c6ba8a4859c37e P ad2afcd7dff742ec8656f3faa395051e: Bootstrap starting.
I20260926 01:10:15.508774 5476 log.cc:824] T 5b87baddc1db4dafb1c6ba8a4859c37e P d8f719bbbc2c4c4aad29dd33860292c6: Log is configured to *not* fsync() on all Append() calls
I20260926 01:10:15.508957 5475 tablet_bootstrap.cc:492] T 5b87baddc1db4dafb1c6ba8a4859c37e P 078c9dbc14de40e289b6aab1df4e64b9: No bootstrap required, opened a new log
I20260926 01:10:15.509025 5475 ts_tablet_manager.cc:1397] T 5b87baddc1db4dafb1c6ba8a4859c37e P 078c9dbc14de40e289b6aab1df4e64b9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 01:10:15.509516 5476 tablet_bootstrap.cc:492] T 5b87baddc1db4dafb1c6ba8a4859c37e P d8f719bbbc2c4c4aad29dd33860292c6: No bootstrap required, opened a new log
I20260926 01:10:15.509579 5476 ts_tablet_manager.cc:1397] T 5b87baddc1db4dafb1c6ba8a4859c37e P d8f719bbbc2c4c4aad29dd33860292c6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 01:10:15.510510 5477 tablet_bootstrap.cc:654] T 5b87baddc1db4dafb1c6ba8a4859c37e P ad2afcd7dff742ec8656f3faa395051e: Neither blocks nor log segments found. Creating new log.
I20260926 01:10:15.510815 5477 log.cc:824] T 5b87baddc1db4dafb1c6ba8a4859c37e P ad2afcd7dff742ec8656f3faa395051e: Log is configured to *not* fsync() on all Append() calls
I20260926 01:10:15.511435 5477 tablet_bootstrap.cc:492] T 5b87baddc1db4dafb1c6ba8a4859c37e P ad2afcd7dff742ec8656f3faa395051e: No bootstrap required, opened a new log
I20260926 01:10:15.511493 5477 ts_tablet_manager.cc:1397] T 5b87baddc1db4dafb1c6ba8a4859c37e P ad2afcd7dff742ec8656f3faa395051e: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260926 01:10:15.511636 5475 raft_consensus.cc:348] T 5b87baddc1db4dafb1c6ba8a4859c37e P 078c9dbc14de40e289b6aab1df4e64b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8f719bbbc2c4c4aad29dd33860292c6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34841 } } peers { permanent_uuid: "078c9dbc14de40e289b6aab1df4e64b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43849 } } peers { permanent_uuid: "ad2afcd7dff742ec8656f3faa395051e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45773 } }
I20260926 01:10:15.511839 5475 raft_consensus.cc:374] T 5b87baddc1db4dafb1c6ba8a4859c37e P 078c9dbc14de40e289b6aab1df4e64b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:10:15.511891 5475 raft_consensus.cc:729] T 5b87baddc1db4dafb1c6ba8a4859c37e P 078c9dbc14de40e289b6aab1df4e64b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 078c9dbc14de40e289b6aab1df4e64b9, State: Initialized, Role: FOLLOWER
I20260926 01:10:15.512010 5475 consensus_queue.cc:260] T 5b87baddc1db4dafb1c6ba8a4859c37e P 078c9dbc14de40e289b6aab1df4e64b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8f719bbbc2c4c4aad29dd33860292c6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34841 } } peers { permanent_uuid: "078c9dbc14de40e289b6aab1df4e64b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43849 } } peers { permanent_uuid: "ad2afcd7dff742ec8656f3faa395051e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45773 } }
I20260926 01:10:15.512166 5476 raft_consensus.cc:348] T 5b87baddc1db4dafb1c6ba8a4859c37e P d8f719bbbc2c4c4aad29dd33860292c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8f719bbbc2c4c4aad29dd33860292c6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34841 } } peers { permanent_uuid: "078c9dbc14de40e289b6aab1df4e64b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43849 } } peers { permanent_uuid: "ad2afcd7dff742ec8656f3faa395051e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45773 } }
I20260926 01:10:15.512315 5199 heartbeater.cc:503] Master 127.0.98.62:45611 was elected leader, sending a full tablet report...
I20260926 01:10:15.512320 5476 raft_consensus.cc:374] T 5b87baddc1db4dafb1c6ba8a4859c37e P d8f719bbbc2c4c4aad29dd33860292c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:10:15.512362 5476 raft_consensus.cc:729] T 5b87baddc1db4dafb1c6ba8a4859c37e P d8f719bbbc2c4c4aad29dd33860292c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8f719bbbc2c4c4aad29dd33860292c6, State: Initialized, Role: FOLLOWER
I20260926 01:10:15.512471 5476 consensus_queue.cc:260] T 5b87baddc1db4dafb1c6ba8a4859c37e P d8f719bbbc2c4c4aad29dd33860292c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8f719bbbc2c4c4aad29dd33860292c6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34841 } } peers { permanent_uuid: "078c9dbc14de40e289b6aab1df4e64b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43849 } } peers { permanent_uuid: "ad2afcd7dff742ec8656f3faa395051e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45773 } }
I20260926 01:10:15.512560 5475 ts_tablet_manager.cc:1428] T 5b87baddc1db4dafb1c6ba8a4859c37e P 078c9dbc14de40e289b6aab1df4e64b9: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260926 01:10:15.512697 5476 ts_tablet_manager.cc:1428] T 5b87baddc1db4dafb1c6ba8a4859c37e P d8f719bbbc2c4c4aad29dd33860292c6: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 01:10:15.512805 5461 heartbeater.cc:503] Master 127.0.98.62:45611 was elected leader, sending a full tablet report...
I20260926 01:10:15.513986 5477 raft_consensus.cc:348] T 5b87baddc1db4dafb1c6ba8a4859c37e P ad2afcd7dff742ec8656f3faa395051e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8f719bbbc2c4c4aad29dd33860292c6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34841 } } peers { permanent_uuid: "078c9dbc14de40e289b6aab1df4e64b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43849 } } peers { permanent_uuid: "ad2afcd7dff742ec8656f3faa395051e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45773 } }
I20260926 01:10:15.514117 5477 raft_consensus.cc:374] T 5b87baddc1db4dafb1c6ba8a4859c37e P ad2afcd7dff742ec8656f3faa395051e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:10:15.514147 5477 raft_consensus.cc:729] T 5b87baddc1db4dafb1c6ba8a4859c37e P ad2afcd7dff742ec8656f3faa395051e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad2afcd7dff742ec8656f3faa395051e, State: Initialized, Role: FOLLOWER
I20260926 01:10:15.514258 5477 consensus_queue.cc:260] T 5b87baddc1db4dafb1c6ba8a4859c37e P ad2afcd7dff742ec8656f3faa395051e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8f719bbbc2c4c4aad29dd33860292c6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34841 } } peers { permanent_uuid: "078c9dbc14de40e289b6aab1df4e64b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43849 } } peers { permanent_uuid: "ad2afcd7dff742ec8656f3faa395051e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45773 } }
I20260926 01:10:15.514474 5477 ts_tablet_manager.cc:1428] T 5b87baddc1db4dafb1c6ba8a4859c37e P ad2afcd7dff742ec8656f3faa395051e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 01:10:15.514534 5330 heartbeater.cc:503] Master 127.0.98.62:45611 was elected leader, sending a full tablet report...
I20260926 01:10:15.576905 5482 raft_consensus.cc:482] T 5b87baddc1db4dafb1c6ba8a4859c37e P d8f719bbbc2c4c4aad29dd33860292c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:10:15.577030 5482 raft_consensus.cc:504] T 5b87baddc1db4dafb1c6ba8a4859c37e P d8f719bbbc2c4c4aad29dd33860292c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8f719bbbc2c4c4aad29dd33860292c6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34841 } } peers { permanent_uuid: "078c9dbc14de40e289b6aab1df4e64b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43849 } } peers { permanent_uuid: "ad2afcd7dff742ec8656f3faa395051e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45773 } }
I20260926 01:10:15.577415 5482 leader_election.cc:288] T 5b87baddc1db4dafb1c6ba8a4859c37e P d8f719bbbc2c4c4aad29dd33860292c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 078c9dbc14de40e289b6aab1df4e64b9 (127.0.98.1:43849), ad2afcd7dff742ec8656f3faa395051e (127.0.98.2:45773)
I20260926 01:10:15.578763 5463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:15.577633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45773 (local address 127.0.98.3:54187)
0926 01:10:15.577731 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:15.577743 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:15.577783 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:15.577978 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:15.577978 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:15.577981 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:10:15.578406 (+ 425us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 01:10:15.578406 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:10:15.578413 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 01:10:15.578426 (+ 13us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:10:15.578619 (+ 193us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:10:15.578620 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:10:15.578620 (+ 0us) client_negotiation.cc:769] Sending connection context
0926 01:10:15.578665 (+ 45us) client_negotiation.cc:240] Negotiation successful
0926 01:10:15.578678 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80}
I20260926 01:10:15.578754 5474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:15.577766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54187 (local address 127.0.98.2:45773)
0926 01:10:15.577811 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:15.577833 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:15.577843 (+ 10us) server_negotiation.cc:407] Connection header received
0926 01:10:15.577872 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:15.577873 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:15.577879 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:15.577954 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:15.578519 (+ 565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:10:15.578521 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:10:15.578521 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:10:15.578526 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:10:15.578564 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:10:15.578569 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:10:15.578621 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:15.578667 (+ 46us) server_negotiation.cc:299] Negotiation successful
0926 01:10:15.578683 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 01:10:15.578830 5473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:15.577766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38613 (local address 127.0.98.1:43849)
0926 01:10:15.577811 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:15.577813 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:15.577961 (+ 148us) server_negotiation.cc:407] Connection header received
0926 01:10:15.578003 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:15.578004 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:15.578008 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:15.578050 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:15.578599 (+ 549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:10:15.578601 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:10:15.578601 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:10:15.578606 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:10:15.578621 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:10:15.578626 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:10:15.578678 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:15.578749 (+ 71us) server_negotiation.cc:299] Negotiation successful
0926 01:10:15.578761 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 01:10:15.579012 5484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:15.577633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43849 (local address 127.0.98.3:38613)
0926 01:10:15.577898 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:15.577906 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:15.577950 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:15.578082 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:15.578082 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:15.578085 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:10:15.578520 (+ 435us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 01:10:15.578520 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:10:15.578525 (+ 5us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 01:10:15.578534 (+ 9us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:10:15.578690 (+ 156us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:10:15.578691 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:10:15.578692 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 01:10:15.578740 (+ 48us) client_negotiation.cc:240] Negotiation successful
0926 01:10:15.578752 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"mutex_wait_us":45,"thread_start_us":88,"threads_started":1}
I20260926 01:10:15.579190 5284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b87baddc1db4dafb1c6ba8a4859c37e" candidate_uuid: "d8f719bbbc2c4c4aad29dd33860292c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad2afcd7dff742ec8656f3faa395051e" is_pre_election: true
I20260926 01:10:15.579373 5284 raft_consensus.cc:2454] T 5b87baddc1db4dafb1c6ba8a4859c37e P ad2afcd7dff742ec8656f3faa395051e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8f719bbbc2c4c4aad29dd33860292c6 in term 0.
I20260926 01:10:15.579344 5153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b87baddc1db4dafb1c6ba8a4859c37e" candidate_uuid: "d8f719bbbc2c4c4aad29dd33860292c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "078c9dbc14de40e289b6aab1df4e64b9" is_pre_election: true
I20260926 01:10:15.579478 5153 raft_consensus.cc:2454] T 5b87baddc1db4dafb1c6ba8a4859c37e P 078c9dbc14de40e289b6aab1df4e64b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8f719bbbc2c4c4aad29dd33860292c6 in term 0.
I20260926 01:10:15.579593 5348 leader_election.cc:302] T 5b87baddc1db4dafb1c6ba8a4859c37e P d8f719bbbc2c4c4aad29dd33860292c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad2afcd7dff742ec8656f3faa395051e, d8f719bbbc2c4c4aad29dd33860292c6; no voters:
I20260926 01:10:15.579715 5482 raft_consensus.cc:2781] T 5b87baddc1db4dafb1c6ba8a4859c37e P d8f719bbbc2c4c4aad29dd33860292c6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:10:15.579783 5482 raft_consensus.cc:482] T 5b87baddc1db4dafb1c6ba8a4859c37e P d8f719bbbc2c4c4aad29dd33860292c6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:10:15.579799 5482 raft_consensus.cc:3037] T 5b87baddc1db4dafb1c6ba8a4859c37e P d8f719bbbc2c4c4aad29dd33860292c6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:10:15.580518 5482 raft_consensus.cc:504] T 5b87baddc1db4dafb1c6ba8a4859c37e P d8f719bbbc2c4c4aad29dd33860292c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8f719bbbc2c4c4aad29dd33860292c6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34841 } } peers { permanent_uuid: "078c9dbc14de40e289b6aab1df4e64b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43849 } } peers { permanent_uuid: "ad2afcd7dff742ec8656f3faa395051e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45773 } }
I20260926 01:10:15.580636 5482 leader_election.cc:288] T 5b87baddc1db4dafb1c6ba8a4859c37e P d8f719bbbc2c4c4aad29dd33860292c6 [CANDIDATE]: Term 1 election: Requested vote from peers 078c9dbc14de40e289b6aab1df4e64b9 (127.0.98.1:43849), ad2afcd7dff742ec8656f3faa395051e (127.0.98.2:45773)
I20260926 01:10:15.580793 5284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b87baddc1db4dafb1c6ba8a4859c37e" candidate_uuid: "d8f719bbbc2c4c4aad29dd33860292c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad2afcd7dff742ec8656f3faa395051e"
I20260926 01:10:15.580878 5284 raft_consensus.cc:3037] T 5b87baddc1db4dafb1c6ba8a4859c37e P ad2afcd7dff742ec8656f3faa395051e [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:10:15.580938 5153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b87baddc1db4dafb1c6ba8a4859c37e" candidate_uuid: "d8f719bbbc2c4c4aad29dd33860292c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "078c9dbc14de40e289b6aab1df4e64b9"
I20260926 01:10:15.581122 5153 raft_consensus.cc:3037] T 5b87baddc1db4dafb1c6ba8a4859c37e P 078c9dbc14de40e289b6aab1df4e64b9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:10:15.581660 5284 raft_consensus.cc:2454] T 5b87baddc1db4dafb1c6ba8a4859c37e P ad2afcd7dff742ec8656f3faa395051e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8f719bbbc2c4c4aad29dd33860292c6 in term 1.
I20260926 01:10:15.581877 5348 leader_election.cc:302] T 5b87baddc1db4dafb1c6ba8a4859c37e P d8f719bbbc2c4c4aad29dd33860292c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad2afcd7dff742ec8656f3faa395051e, d8f719bbbc2c4c4aad29dd33860292c6; no voters:
I20260926 01:10:15.581967 5482 raft_consensus.cc:2781] T 5b87baddc1db4dafb1c6ba8a4859c37e P d8f719bbbc2c4c4aad29dd33860292c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:10:15.582115 5482 raft_consensus.cc:686] T 5b87baddc1db4dafb1c6ba8a4859c37e P d8f719bbbc2c4c4aad29dd33860292c6 [term 1 LEADER]: Becoming Leader. State: Replica: d8f719bbbc2c4c4aad29dd33860292c6, State: Running, Role: LEADER
I20260926 01:10:15.582207 5482 consensus_queue.cc:237] T 5b87baddc1db4dafb1c6ba8a4859c37e P d8f719bbbc2c4c4aad29dd33860292c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8f719bbbc2c4c4aad29dd33860292c6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34841 } } peers { permanent_uuid: "078c9dbc14de40e289b6aab1df4e64b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43849 } } peers { permanent_uuid: "ad2afcd7dff742ec8656f3faa395051e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45773 } }
I20260926 01:10:15.582597 5153 raft_consensus.cc:2454] T 5b87baddc1db4dafb1c6ba8a4859c37e P 078c9dbc14de40e289b6aab1df4e64b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8f719bbbc2c4c4aad29dd33860292c6 in term 1.
I20260926 01:10:15.582991 5010 catalog_manager.cc:5495] T 5b87baddc1db4dafb1c6ba8a4859c37e P d8f719bbbc2c4c4aad29dd33860292c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to d8f719bbbc2c4c4aad29dd33860292c6 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "d8f719bbbc2c4c4aad29dd33860292c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8f719bbbc2c4c4aad29dd33860292c6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34841 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "078c9dbc14de40e289b6aab1df4e64b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad2afcd7dff742ec8656f3faa395051e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45773 } health_report { overall_health: UNKNOWN } } }
I20260926 01:10:15.598763 5472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:15.597704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35474 (local address 127.0.98.3:34841)
0926 01:10:15.597756 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:15.597758 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:15.597767 (+ 9us) server_negotiation.cc:407] Connection header received
0926 01:10:15.597851 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:15.597852 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:15.597873 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:15.597922 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:15.598536 (+ 614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:10:15.598537 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:10:15.598537 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:10:15.598541 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:10:15.598558 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:10:15.598562 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:10:15.598614 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:15.598673 (+ 59us) server_negotiation.cc:299] Negotiation successful
0926 01:10:15.598686 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260926 01:10:15.600912 5153 raft_consensus.cc:1261] T 5b87baddc1db4dafb1c6ba8a4859c37e P 078c9dbc14de40e289b6aab1df4e64b9 [term 1 FOLLOWER]: Refusing update from remote peer d8f719bbbc2c4c4aad29dd33860292c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:10:15.600921 5284 raft_consensus.cc:1261] T 5b87baddc1db4dafb1c6ba8a4859c37e P ad2afcd7dff742ec8656f3faa395051e [term 1 FOLLOWER]: Refusing update from remote peer d8f719bbbc2c4c4aad29dd33860292c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:10:15.601308 5485 consensus_queue.cc:1035] T 5b87baddc1db4dafb1c6ba8a4859c37e P d8f719bbbc2c4c4aad29dd33860292c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "078c9dbc14de40e289b6aab1df4e64b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:10:15.601475 5482 consensus_queue.cc:1035] T 5b87baddc1db4dafb1c6ba8a4859c37e P d8f719bbbc2c4c4aad29dd33860292c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad2afcd7dff742ec8656f3faa395051e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:10:15.603888 5488 mvcc.cc:205] Tried to move back new op lower bound from 7333417023898124288 to 7333417023825395712. Current Snapshot: MvccSnapshot[applied={T|T < 7333417023898124288}]
I20260926 01:10:15.603981 5489 mvcc.cc:205] Tried to move back new op lower bound from 7333417023898124288 to 7333417023825395712. Current Snapshot: MvccSnapshot[applied={T|T < 7333417023898124288}]
I20260926 01:10:15.607105 5010 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47426:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:10:15.607239 5010 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47426:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:10:15.607988 5010 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3b3da3a1543b47d8997e19c61ff531fe: Sending DeleteTablet for 3 replicas of tablet 5b87baddc1db4dafb1c6ba8a4859c37e
I20260926 01:10:15.608323 5395 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b87baddc1db4dafb1c6ba8a4859c37e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:10:15 UTC) from {username='slave'} at 127.0.0.1:35472
I20260926 01:10:15.608349 5264 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b87baddc1db4dafb1c6ba8a4859c37e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:10:15 UTC) from {username='slave'} at 127.0.0.1:59512
I20260926 01:10:15.608382 5133 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b87baddc1db4dafb1c6ba8a4859c37e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:10:15 UTC) from {username='slave'} at 127.0.0.1:60420
I20260926 01:10:15.608700 5499 tablet_replica.cc:330] T 5b87baddc1db4dafb1c6ba8a4859c37e P ad2afcd7dff742ec8656f3faa395051e: stopping tablet replica
I20260926 01:10:15.608705 5500 tablet_replica.cc:330] T 5b87baddc1db4dafb1c6ba8a4859c37e P 078c9dbc14de40e289b6aab1df4e64b9: stopping tablet replica
I20260926 01:10:15.608816 5500 raft_consensus.cc:2229] T 5b87baddc1db4dafb1c6ba8a4859c37e P 078c9dbc14de40e289b6aab1df4e64b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:10:15.608906 5499 raft_consensus.cc:2229] T 5b87baddc1db4dafb1c6ba8a4859c37e P ad2afcd7dff742ec8656f3faa395051e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:10:15.608937 5500 raft_consensus.cc:2258] T 5b87baddc1db4dafb1c6ba8a4859c37e P 078c9dbc14de40e289b6aab1df4e64b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:10:15.608956 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 5071
I20260926 01:10:15.609567 5499 raft_consensus.cc:2258] T 5b87baddc1db4dafb1c6ba8a4859c37e P ad2afcd7dff742ec8656f3faa395051e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:10:15.609985 5498 tablet_replica.cc:330] T 5b87baddc1db4dafb1c6ba8a4859c37e P d8f719bbbc2c4c4aad29dd33860292c6: stopping tablet replica
I20260926 01:10:15.610149 5498 raft_consensus.cc:2229] T 5b87baddc1db4dafb1c6ba8a4859c37e P d8f719bbbc2c4c4aad29dd33860292c6 [term 1 LEADER]: Raft consensus shutting down.
I20260926 01:10:15.610332 5498 raft_consensus.cc:2258] T 5b87baddc1db4dafb1c6ba8a4859c37e P d8f719bbbc2c4c4aad29dd33860292c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:10:15.610361 5499 ts_tablet_manager.cc:1905] T 5b87baddc1db4dafb1c6ba8a4859c37e P ad2afcd7dff742ec8656f3faa395051e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:10:15.610844 5498 ts_tablet_manager.cc:1905] T 5b87baddc1db4dafb1c6ba8a4859c37e P d8f719bbbc2c4c4aad29dd33860292c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:10:15.612193 5499 ts_tablet_manager.cc:1918] T 5b87baddc1db4dafb1c6ba8a4859c37e P ad2afcd7dff742ec8656f3faa395051e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:10:15.612278 5499 log.cc:1196] T 5b87baddc1db4dafb1c6ba8a4859c37e P ad2afcd7dff742ec8656f3faa395051e: Deleting WAL directory at /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384993574762-392-0/minicluster-data/ts-1/wal/wals/5b87baddc1db4dafb1c6ba8a4859c37e
I20260926 01:10:15.612545 5499 ts_tablet_manager.cc:1939] T 5b87baddc1db4dafb1c6ba8a4859c37e P ad2afcd7dff742ec8656f3faa395051e: Deleting consensus metadata
I20260926 01:10:15.612568 5498 ts_tablet_manager.cc:1918] T 5b87baddc1db4dafb1c6ba8a4859c37e P d8f719bbbc2c4c4aad29dd33860292c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:10:15.612754 5498 log.cc:1196] T 5b87baddc1db4dafb1c6ba8a4859c37e P d8f719bbbc2c4c4aad29dd33860292c6: Deleting WAL directory at /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384993574762-392-0/minicluster-data/ts-2/wal/wals/5b87baddc1db4dafb1c6ba8a4859c37e
I20260926 01:10:15.613019 4995 catalog_manager.cc:4841] TS ad2afcd7dff742ec8656f3faa395051e (127.0.98.2:45773): tablet 5b87baddc1db4dafb1c6ba8a4859c37e (table test-table [id=4f6321bad73a486596d8a76191fb21d2]) successfully deleted
I20260926 01:10:15.613063 5498 ts_tablet_manager.cc:1939] T 5b87baddc1db4dafb1c6ba8a4859c37e P d8f719bbbc2c4c4aad29dd33860292c6: Deleting consensus metadata
I20260926 01:10:15.613472 4996 catalog_manager.cc:4841] TS d8f719bbbc2c4c4aad29dd33860292c6 (127.0.98.3:34841): tablet 5b87baddc1db4dafb1c6ba8a4859c37e (table test-table [id=4f6321bad73a486596d8a76191fb21d2]) successfully deleted
W20260926 01:10:15.616532 4998 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43849 (error 108)
I20260926 01:10:15.617085 5471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:15.616926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43849 (local address 127.0.0.1:60432)
0926 01:10:15.617005 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:15.617054 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260926 01:10:15.617237 4998 catalog_manager.cc:4584] TS 078c9dbc14de40e289b6aab1df4e64b9 (127.0.98.1:43849): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5b87baddc1db4dafb1c6ba8a4859c37e: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43849: connect: Connection refused (error 111)
I20260926 01:10:15.617455 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 5202
I20260926 01:10:15.622278 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 5333
I20260926 01:10:15.626480 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 4980
2026-09-26T01:10:15Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (676 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260926 01:10:15.645228 392 mini_oidc.cc:152] Starting JWKS server
I20260926 01:10:15.646365 392 webserver.cc:460] Webserver started at https://127.0.0.1:41833/ using document root <none> and password file <none>
I20260926 01:10:15.646461 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 01:10:15.646621 392 webserver.cc:460] Webserver started at http://127.0.0.1:37573/ using document root <none> and password file <none>
2026-09-26T01:10:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:10:15Z Disabled control of system clock
I20260926 01:10:15.655149 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790384993574762-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790384993574762-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790384993574762-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40715
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:42851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40715
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41833/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790384993574762-392-0/testchainca.cert with env {}
W20260926 01:10:15.738642 5507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:10:15.738889 5507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:10:15.738914 5507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:10:15.740309 5507 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:10:15.740367 5507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:10:15.740379 5507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:10:15.740391 5507 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:10:15.740402 5507 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 01:10:15.740413 5507 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 01:10:15.740423 5507 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41833/jwks.json
I20260926 01:10:15.742206 5507 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790384993574762-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790384993574762-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40715
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40715
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41833/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790384993574762-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5507
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790384993574762-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:10:15.742479 5507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:10:15.742708 5507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:10:15.747628 5513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:10:15.747645 5507 server_base.cc:1034] running on GCE node
W20260926 01:10:15.747649 5512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:10:15.747628 5515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:10:15.748044 5507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:10:15.748323 5507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:10:15.749507 5507 hybrid_clock.cc:648] HybridClock initialized: now 1790385015749481 us; error 39 us; skew 500 ppm
I20260926 01:10:15.756000 5507 webserver.cc:460] Webserver started at http://127.0.98.62:42031/ using document root <none> and password file <none>
I20260926 01:10:15.756304 5507 fs_manager.cc:359] Metadata directory not provided
I20260926 01:10:15.756361 5507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:10:15.756487 5507 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:10:15.757599 5507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790384993574762-392-0/minicluster-data/master-0/data/instance:
uuid: "982a75662b834a3d895d8f6890e9bd1c"
format_stamp: "Formatted at 2026-09-26 01:10:15 on dist-test-slave-7b22"
server_key: "e69cffa6a7e4610151eeaf082faf810d"
server_key_iv: "27e380bfbb9b0c2303a5531b47cdd722"
server_key_version: "encryptionkey@0"
I20260926 01:10:15.757974 5507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790384993574762-392-0/minicluster-data/master-0/wal/instance:
uuid: "982a75662b834a3d895d8f6890e9bd1c"
format_stamp: "Formatted at 2026-09-26 01:10:15 on dist-test-slave-7b22"
server_key: "e69cffa6a7e4610151eeaf082faf810d"
server_key_iv: "27e380bfbb9b0c2303a5531b47cdd722"
server_key_version: "encryptionkey@0"
I20260926 01:10:15.759300 5507 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 01:10:15.760191 5521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:15.760368 5507 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 01:10:15.760465 5507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790384993574762-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790384993574762-392-0/minicluster-data/master-0/wal
uuid: "982a75662b834a3d895d8f6890e9bd1c"
format_stamp: "Formatted at 2026-09-26 01:10:15 on dist-test-slave-7b22"
server_key: "e69cffa6a7e4610151eeaf082faf810d"
server_key_iv: "27e380bfbb9b0c2303a5531b47cdd722"
server_key_version: "encryptionkey@0"
I20260926 01:10:15.760532 5507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790384993574762-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790384993574762-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790384993574762-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:10:15.774957 5507 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:10:15.775288 5507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:10:15.775424 5507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:10:15.779474 5507 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40715
I20260926 01:10:15.779526 5573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40715 every 8 connection(s)
I20260926 01:10:15.779866 5507 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
I20260926 01:10:15.780244 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 5507
I20260926 01:10:15.780368 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790384993574762-392-0/minicluster-data/master-0/wal/instance
I20260926 01:10:15.780738 392 external_mini_cluster.cc:1428] Setting key ccb6d58c8dce4b2b7bc485220585ab27
I20260926 01:10:15.780901 5574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:10:15.785125 5574 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 982a75662b834a3d895d8f6890e9bd1c: Bootstrap starting.
I20260926 01:10:15.786201 5574 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 982a75662b834a3d895d8f6890e9bd1c: Neither blocks nor log segments found. Creating new log.
I20260926 01:10:15.786562 5574 log.cc:824] T 00000000000000000000000000000000 P 982a75662b834a3d895d8f6890e9bd1c: Log is configured to *not* fsync() on all Append() calls
I20260926 01:10:15.787597 5574 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 982a75662b834a3d895d8f6890e9bd1c: No bootstrap required, opened a new log
I20260926 01:10:15.787956 5577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:15.781327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44054 (local address 127.0.98.62:40715)
0926 01:10:15.781545 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:15.781554 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:15.781589 (+ 35us) server_negotiation.cc:407] Connection header received
0926 01:10:15.782223 (+ 634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:15.782233 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:15.782323 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:15.782486 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:15.782770 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:15.784105 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:15.785428 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:15.786056 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:15.786218 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:15.787048 (+ 830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:10:15.787054 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:10:15.787054 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:10:15.787373 (+ 319us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:10:15.787417 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:10:15.787433 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:10:15.787502 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:15.787639 (+ 137us) server_negotiation.cc:299] Negotiation successful
0926 01:10:15.787747 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":90,"threads_started":1}
I20260926 01:10:15.789166 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 01:10:15.789992 5574 raft_consensus.cc:348] T 00000000000000000000000000000000 P 982a75662b834a3d895d8f6890e9bd1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "982a75662b834a3d895d8f6890e9bd1c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40715 } }
I20260926 01:10:15.790174 5574 raft_consensus.cc:374] T 00000000000000000000000000000000 P 982a75662b834a3d895d8f6890e9bd1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:10:15.790220 5574 raft_consensus.cc:729] T 00000000000000000000000000000000 P 982a75662b834a3d895d8f6890e9bd1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 982a75662b834a3d895d8f6890e9bd1c, State: Initialized, Role: FOLLOWER
I20260926 01:10:15.790351 5574 consensus_queue.cc:260] T 00000000000000000000000000000000 P 982a75662b834a3d895d8f6890e9bd1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "982a75662b834a3d895d8f6890e9bd1c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40715 } }
I20260926 01:10:15.790438 5574 raft_consensus.cc:388] T 00000000000000000000000000000000 P 982a75662b834a3d895d8f6890e9bd1c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:10:15.790478 5574 raft_consensus.cc:482] T 00000000000000000000000000000000 P 982a75662b834a3d895d8f6890e9bd1c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:10:15.790526 5574 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 982a75662b834a3d895d8f6890e9bd1c [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:10:15.791142 5574 raft_consensus.cc:504] T 00000000000000000000000000000000 P 982a75662b834a3d895d8f6890e9bd1c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "982a75662b834a3d895d8f6890e9bd1c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40715 } }
I20260926 01:10:15.791239 5574 leader_election.cc:302] T 00000000000000000000000000000000 P 982a75662b834a3d895d8f6890e9bd1c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 982a75662b834a3d895d8f6890e9bd1c; no voters:
I20260926 01:10:15.791399 5574 leader_election.cc:288] T 00000000000000000000000000000000 P 982a75662b834a3d895d8f6890e9bd1c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:10:15.791467 5579 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 982a75662b834a3d895d8f6890e9bd1c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:10:15.791608 5579 raft_consensus.cc:686] T 00000000000000000000000000000000 P 982a75662b834a3d895d8f6890e9bd1c [term 1 LEADER]: Becoming Leader. State: Replica: 982a75662b834a3d895d8f6890e9bd1c, State: Running, Role: LEADER
I20260926 01:10:15.791661 5574 sys_catalog.cc:564] T 00000000000000000000000000000000 P 982a75662b834a3d895d8f6890e9bd1c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:10:15.791697 5579 consensus_queue.cc:237] T 00000000000000000000000000000000 P 982a75662b834a3d895d8f6890e9bd1c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "982a75662b834a3d895d8f6890e9bd1c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40715 } }
I20260926 01:10:15.792043 5581 sys_catalog.cc:455] T 00000000000000000000000000000000 P 982a75662b834a3d895d8f6890e9bd1c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 982a75662b834a3d895d8f6890e9bd1c. Latest consensus state: current_term: 1 leader_uuid: "982a75662b834a3d895d8f6890e9bd1c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "982a75662b834a3d895d8f6890e9bd1c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40715 } } }
I20260926 01:10:15.792055 5580 sys_catalog.cc:455] T 00000000000000000000000000000000 P 982a75662b834a3d895d8f6890e9bd1c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "982a75662b834a3d895d8f6890e9bd1c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "982a75662b834a3d895d8f6890e9bd1c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40715 } } }
I20260926 01:10:15.792140 5580 sys_catalog.cc:458] T 00000000000000000000000000000000 P 982a75662b834a3d895d8f6890e9bd1c [sys.catalog]: This master's current role is: LEADER
I20260926 01:10:15.792136 5581 sys_catalog.cc:458] T 00000000000000000000000000000000 P 982a75662b834a3d895d8f6890e9bd1c [sys.catalog]: This master's current role is: LEADER
I20260926 01:10:15.792426 5586 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:10:15.793081 5586 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:10:15.794561 5586 catalog_manager.cc:1347] Generated new cluster ID: b05885bcb9d545158e7798b09a17d06d
I20260926 01:10:15.794615 5586 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 01:10:15.808037 5586 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:10:15.808812 5586 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:10:15.818049 5586 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 982a75662b834a3d895d8f6890e9bd1c: Generated new TSK 0
I20260926 01:10:15.818280 5586 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260926 01:10:15.853075 5604 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260926 01:10:15.853171 5604 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 01:10:15.853540 5577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:15.833205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44062 (local address 127.0.98.62:40715)
0926 01:10:15.833258 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:15.833260 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:15.833404 (+ 144us) server_negotiation.cc:407] Connection header received
0926 01:10:15.833501 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:15.833502 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:15.833514 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:15.833579 (+ 65us) server_negotiation.cc:226] Negotiated authn=JWT
0926 01:10:15.833768 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:15.834368 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:15.835506 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:15.835695 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:15.835754 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:15.836043 (+ 289us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 01:10:15.853239 (+ 17196us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41833/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0926 01:10:15.853443 (+ 204us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44062: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41833/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":28}
W20260926 01:10:15.853598 5577 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44062: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41833/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260926 01:10:15.853873 5603 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:10:15.833102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40715 (local address 127.0.0.1:44062)
0926 01:10:15.833309 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:15.833321 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:15.833402 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:15.833597 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:15.833598 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:15.833599 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0926 01:10:15.833678 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:15.833680 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:15.834440 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:15.834441 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:15.835409 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:15.835412 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:15.835762 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:15.835766 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:15.835831 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:15.835858 (+ 27us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 01:10:15.853419 (+ 17561us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41833/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0926 01:10:15.853530 (+ 111us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:40715: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41833/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":101,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.98.62:40715: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41833/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260926 01:10:15.854313 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 5507
2026-09-26T01:10:15Z chronyd exiting
I20260926 01:10:15.890303 392 test_util.cc:175] -----------------------------------------------
I20260926 01:10:15.890403 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790384993574762-392-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (246 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260926 01:10:15.891227 392 mini_oidc.cc:152] Starting JWKS server
I20260926 01:10:15.891923 392 webserver.cc:460] Webserver started at https://127.0.0.1:46873/ using document root <none> and password file <none>
I20260926 01:10:15.891968 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 01:10:15.892105 392 webserver.cc:460] Webserver started at http://127.0.0.1:41051/ using document root <none> and password file <none>
2026-09-26T01:10:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:10:15Z Disabled control of system clock
I20260926 01:10:15.899506 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790384993574762-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790384993574762-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790384993574762-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46237
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:36821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46237
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46873/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790384993574762-392-0/testchainca.cert with env {}
W20260926 01:10:15.979542 5610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:10:15.979789 5610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:10:15.979820 5610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:10:15.981066 5610 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:10:15.981108 5610 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:10:15.981123 5610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:10:15.981141 5610 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:10:15.981155 5610 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 01:10:15.981173 5610 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 01:10:15.981195 5610 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46873/jwks.json
I20260926 01:10:15.982824 5610 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790384993574762-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790384993574762-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46237
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46237
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46873/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790384993574762-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5610
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790384993574762-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:10:15.983074 5610 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:10:15.983278 5610 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:10:15.987918 5616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:10:15.987977 5615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:10:15.988121 5610 server_base.cc:1034] running on GCE node
W20260926 01:10:15.988067 5618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:10:15.988428 5610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:10:15.988711 5610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:10:15.989866 5610 hybrid_clock.cc:648] HybridClock initialized: now 1790385015989845 us; error 34 us; skew 500 ppm
I20260926 01:10:15.991067 5610 webserver.cc:460] Webserver started at http://127.0.98.62:42513/ using document root <none> and password file <none>
I20260926 01:10:15.991284 5610 fs_manager.cc:359] Metadata directory not provided
I20260926 01:10:15.991331 5610 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:10:15.991436 5610 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:10:15.992540 5610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790384993574762-392-0/minicluster-data/master-0/data/instance:
uuid: "706640be382b4bd8bb54bc3699823827"
format_stamp: "Formatted at 2026-09-26 01:10:15 on dist-test-slave-7b22"
server_key: "8dd016745d9b8ab49407c15459f9faa9"
server_key_iv: "9a12a7325c06828bd5c477a05c54c0b5"
server_key_version: "encryptionkey@0"
I20260926 01:10:15.992908 5610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790384993574762-392-0/minicluster-data/master-0/wal/instance:
uuid: "706640be382b4bd8bb54bc3699823827"
format_stamp: "Formatted at 2026-09-26 01:10:15 on dist-test-slave-7b22"
server_key: "8dd016745d9b8ab49407c15459f9faa9"
server_key_iv: "9a12a7325c06828bd5c477a05c54c0b5"
server_key_version: "encryptionkey@0"
I20260926 01:10:15.994447 5610 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 01:10:15.995185 5624 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:15.995325 5610 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 01:10:15.995405 5610 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790384993574762-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790384993574762-392-0/minicluster-data/master-0/wal
uuid: "706640be382b4bd8bb54bc3699823827"
format_stamp: "Formatted at 2026-09-26 01:10:15 on dist-test-slave-7b22"
server_key: "8dd016745d9b8ab49407c15459f9faa9"
server_key_iv: "9a12a7325c06828bd5c477a05c54c0b5"
server_key_version: "encryptionkey@0"
I20260926 01:10:15.995471 5610 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790384993574762-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790384993574762-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790384993574762-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:10:16.014051 5610 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:10:16.014405 5610 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:10:16.014554 5610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:10:16.018326 5610 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46237
I20260926 01:10:16.018414 5676 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46237 every 8 connection(s)
I20260926 01:10:16.018729 5610 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
I20260926 01:10:16.020078 5677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:10:16.022604 5677 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 706640be382b4bd8bb54bc3699823827: Bootstrap starting.
I20260926 01:10:16.023211 5677 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 706640be382b4bd8bb54bc3699823827: Neither blocks nor log segments found. Creating new log.
I20260926 01:10:16.023473 5677 log.cc:824] T 00000000000000000000000000000000 P 706640be382b4bd8bb54bc3699823827: Log is configured to *not* fsync() on all Append() calls
I20260926 01:10:16.023787 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 5610
I20260926 01:10:16.023876 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790384993574762-392-0/minicluster-data/master-0/wal/instance
I20260926 01:10:16.024040 392 external_mini_cluster.cc:1428] Setting key a7fa3c5e77b1a09ebe2deb7e73d3d083
I20260926 01:10:16.024284 5677 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 706640be382b4bd8bb54bc3699823827: No bootstrap required, opened a new log
I20260926 01:10:16.027083 5677 raft_consensus.cc:348] T 00000000000000000000000000000000 P 706640be382b4bd8bb54bc3699823827 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "706640be382b4bd8bb54bc3699823827" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46237 } }
I20260926 01:10:16.027285 5677 raft_consensus.cc:374] T 00000000000000000000000000000000 P 706640be382b4bd8bb54bc3699823827 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:10:16.027315 5677 raft_consensus.cc:729] T 00000000000000000000000000000000 P 706640be382b4bd8bb54bc3699823827 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 706640be382b4bd8bb54bc3699823827, State: Initialized, Role: FOLLOWER
I20260926 01:10:16.027452 5677 consensus_queue.cc:260] T 00000000000000000000000000000000 P 706640be382b4bd8bb54bc3699823827 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "706640be382b4bd8bb54bc3699823827" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46237 } }
I20260926 01:10:16.027531 5677 raft_consensus.cc:388] T 00000000000000000000000000000000 P 706640be382b4bd8bb54bc3699823827 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:10:16.027563 5677 raft_consensus.cc:482] T 00000000000000000000000000000000 P 706640be382b4bd8bb54bc3699823827 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:10:16.027601 5677 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 706640be382b4bd8bb54bc3699823827 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:10:16.028447 5677 raft_consensus.cc:504] T 00000000000000000000000000000000 P 706640be382b4bd8bb54bc3699823827 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "706640be382b4bd8bb54bc3699823827" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46237 } }
I20260926 01:10:16.028551 5677 leader_election.cc:302] T 00000000000000000000000000000000 P 706640be382b4bd8bb54bc3699823827 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 706640be382b4bd8bb54bc3699823827; no voters:
I20260926 01:10:16.028717 5677 leader_election.cc:288] T 00000000000000000000000000000000 P 706640be382b4bd8bb54bc3699823827 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:10:16.028765 5682 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 706640be382b4bd8bb54bc3699823827 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:10:16.028926 5682 raft_consensus.cc:686] T 00000000000000000000000000000000 P 706640be382b4bd8bb54bc3699823827 [term 1 LEADER]: Becoming Leader. State: Replica: 706640be382b4bd8bb54bc3699823827, State: Running, Role: LEADER
I20260926 01:10:16.028968 5677 sys_catalog.cc:564] T 00000000000000000000000000000000 P 706640be382b4bd8bb54bc3699823827 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:10:16.029029 5682 consensus_queue.cc:237] T 00000000000000000000000000000000 P 706640be382b4bd8bb54bc3699823827 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "706640be382b4bd8bb54bc3699823827" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46237 } }
I20260926 01:10:16.029433 5682 sys_catalog.cc:455] T 00000000000000000000000000000000 P 706640be382b4bd8bb54bc3699823827 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 706640be382b4bd8bb54bc3699823827. Latest consensus state: current_term: 1 leader_uuid: "706640be382b4bd8bb54bc3699823827" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "706640be382b4bd8bb54bc3699823827" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46237 } } }
I20260926 01:10:16.029505 5682 sys_catalog.cc:458] T 00000000000000000000000000000000 P 706640be382b4bd8bb54bc3699823827 [sys.catalog]: This master's current role is: LEADER
I20260926 01:10:16.029553 5683 sys_catalog.cc:455] T 00000000000000000000000000000000 P 706640be382b4bd8bb54bc3699823827 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "706640be382b4bd8bb54bc3699823827" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "706640be382b4bd8bb54bc3699823827" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46237 } } }
I20260926 01:10:16.029616 5683 sys_catalog.cc:458] T 00000000000000000000000000000000 P 706640be382b4bd8bb54bc3699823827 [sys.catalog]: This master's current role is: LEADER
I20260926 01:10:16.029950 5688 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:10:16.030500 5688 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:10:16.031129 5681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:16.024519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48222 (local address 127.0.98.62:46237)
0926 01:10:16.024753 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:16.024763 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:16.024799 (+ 36us) server_negotiation.cc:407] Connection header received
0926 01:10:16.025560 (+ 761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:16.025570 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:16.025652 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:16.025843 (+ 191us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:16.026070 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:16.027450 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:16.028594 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:16.029330 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:16.029799 (+ 469us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:16.030212 (+ 413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:10:16.030216 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:10:16.030216 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:10:16.030412 (+ 196us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:10:16.030454 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:10:16.030470 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:10:16.030802 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:16.030835 (+ 33us) server_negotiation.cc:299] Negotiation successful
0926 01:10:16.030920 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":90,"threads_started":1}
I20260926 01:10:16.032039 5688 catalog_manager.cc:1347] Generated new cluster ID: c221383018d1445c80cc47f1f5ccae52
I20260926 01:10:16.032084 5688 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 01:10:16.032347 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 01:10:16.043799 5688 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:10:16.044345 5688 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:10:16.051528 5688 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 706640be382b4bd8bb54bc3699823827: Generated new TSK 0
I20260926 01:10:16.051716 5688 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260926 01:10:16.078418 5707 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260926 01:10:16.078522 5707 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 01:10:16.078824 5681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:16.059782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48232 (local address 127.0.98.62:46237)
0926 01:10:16.059825 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:16.059827 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:16.059906 (+ 79us) server_negotiation.cc:407] Connection header received
0926 01:10:16.059971 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:16.059972 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:16.059982 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:16.060039 (+ 57us) server_negotiation.cc:226] Negotiated authn=JWT
0926 01:10:16.060179 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:16.060677 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:16.061770 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:16.062039 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:16.062118 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:16.062292 (+ 174us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 01:10:16.078634 (+ 16342us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46873/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0926 01:10:16.078769 (+ 135us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48232: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46873/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":20}
W20260926 01:10:16.078869 5681 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48232: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46873/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260926 01:10:16.079063 5706 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:10:16.059680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46237 (local address 127.0.0.1:48232)
0926 01:10:16.059823 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:16.059836 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:16.059905 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:16.060049 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:16.060050 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:16.060052 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0926 01:10:16.060122 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:16.060124 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:16.060746 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:16.060747 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:16.061687 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:16.061690 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:16.062114 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:16.062118 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:16.062188 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:16.062194 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 01:10:16.078783 (+ 16589us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46873/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0926 01:10:16.078937 (+ 154us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:46237: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46873/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":75,"threads_started":1}
I20260926 01:10:16.079344 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 5610
2026-09-26T01:10:16Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (225 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260926 01:10:16.117379 392 mini_oidc.cc:152] Starting JWKS server
I20260926 01:10:16.118150 392 webserver.cc:460] Webserver started at https://127.0.0.1:42845/ using document root <none> and password file <none>
I20260926 01:10:16.118211 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 01:10:16.118348 392 webserver.cc:460] Webserver started at http://127.0.0.1:33829/ using document root <none> and password file <none>
2026-09-26T01:10:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:10:16Z Disabled control of system clock
I20260926 01:10:16.126801 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790384993574762-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790384993574762-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790384993574762-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33683
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:42707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33683
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42845/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260926 01:10:16.212271 5713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:10:16.212498 5713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:10:16.212530 5713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:10:16.214360 5713 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:10:16.214421 5713 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:10:16.214433 5713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:10:16.214445 5713 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:10:16.214457 5713 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 01:10:16.214468 5713 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 01:10:16.214478 5713 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42845/jwks.json
I20260926 01:10:16.216112 5713 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790384993574762-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790384993574762-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33683
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33683
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42845/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5713
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790384993574762-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:10:16.216388 5713 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:10:16.216696 5713 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:10:16.222025 5721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:10:16.222044 5719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:10:16.222078 5718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:10:16.222129 5713 server_base.cc:1034] running on GCE node
I20260926 01:10:16.222594 5713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:10:16.222894 5713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:10:16.224107 5713 hybrid_clock.cc:648] HybridClock initialized: now 1790385016224081 us; error 38 us; skew 500 ppm
I20260926 01:10:16.225700 5713 webserver.cc:460] Webserver started at http://127.0.98.62:46091/ using document root <none> and password file <none>
I20260926 01:10:16.225997 5713 fs_manager.cc:359] Metadata directory not provided
I20260926 01:10:16.226054 5713 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:10:16.226187 5713 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:10:16.227370 5713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790384993574762-392-0/minicluster-data/master-0/data/instance:
uuid: "dc39be43155a4a5a900c7d3e9680e1c9"
format_stamp: "Formatted at 2026-09-26 01:10:16 on dist-test-slave-7b22"
server_key: "bd5cfa347ef8ef24e6b22f2026fc2a4e"
server_key_iv: "ae0736b597a4798c6b36cacf45940ec8"
server_key_version: "encryptionkey@0"
I20260926 01:10:16.227731 5713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790384993574762-392-0/minicluster-data/master-0/wal/instance:
uuid: "dc39be43155a4a5a900c7d3e9680e1c9"
format_stamp: "Formatted at 2026-09-26 01:10:16 on dist-test-slave-7b22"
server_key: "bd5cfa347ef8ef24e6b22f2026fc2a4e"
server_key_iv: "ae0736b597a4798c6b36cacf45940ec8"
server_key_version: "encryptionkey@0"
I20260926 01:10:16.229092 5713 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 01:10:16.229998 5727 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:16.230141 5713 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 01:10:16.230228 5713 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790384993574762-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790384993574762-392-0/minicluster-data/master-0/wal
uuid: "dc39be43155a4a5a900c7d3e9680e1c9"
format_stamp: "Formatted at 2026-09-26 01:10:16 on dist-test-slave-7b22"
server_key: "bd5cfa347ef8ef24e6b22f2026fc2a4e"
server_key_iv: "ae0736b597a4798c6b36cacf45940ec8"
server_key_version: "encryptionkey@0"
I20260926 01:10:16.230298 5713 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790384993574762-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790384993574762-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790384993574762-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:10:16.239300 5713 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:10:16.239612 5713 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:10:16.239750 5713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:10:16.243724 5713 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33683
I20260926 01:10:16.243857 5779 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33683 every 8 connection(s)
I20260926 01:10:16.244108 5713 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
I20260926 01:10:16.244810 5780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:10:16.247524 5780 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc39be43155a4a5a900c7d3e9680e1c9: Bootstrap starting.
I20260926 01:10:16.248097 5780 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dc39be43155a4a5a900c7d3e9680e1c9: Neither blocks nor log segments found. Creating new log.
I20260926 01:10:16.248382 5780 log.cc:824] T 00000000000000000000000000000000 P dc39be43155a4a5a900c7d3e9680e1c9: Log is configured to *not* fsync() on all Append() calls
I20260926 01:10:16.249282 5780 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc39be43155a4a5a900c7d3e9680e1c9: No bootstrap required, opened a new log
I20260926 01:10:16.251013 5780 raft_consensus.cc:348] T 00000000000000000000000000000000 P dc39be43155a4a5a900c7d3e9680e1c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc39be43155a4a5a900c7d3e9680e1c9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33683 } }
I20260926 01:10:16.251163 5780 raft_consensus.cc:374] T 00000000000000000000000000000000 P dc39be43155a4a5a900c7d3e9680e1c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:10:16.251185 5780 raft_consensus.cc:729] T 00000000000000000000000000000000 P dc39be43155a4a5a900c7d3e9680e1c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc39be43155a4a5a900c7d3e9680e1c9, State: Initialized, Role: FOLLOWER
I20260926 01:10:16.251322 5780 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc39be43155a4a5a900c7d3e9680e1c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc39be43155a4a5a900c7d3e9680e1c9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33683 } }
I20260926 01:10:16.251389 5780 raft_consensus.cc:388] T 00000000000000000000000000000000 P dc39be43155a4a5a900c7d3e9680e1c9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:10:16.251417 5780 raft_consensus.cc:482] T 00000000000000000000000000000000 P dc39be43155a4a5a900c7d3e9680e1c9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:10:16.251466 5780 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dc39be43155a4a5a900c7d3e9680e1c9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:10:16.251461 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 5713
I20260926 01:10:16.251559 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790384993574762-392-0/minicluster-data/master-0/wal/instance
I20260926 01:10:16.251744 392 external_mini_cluster.cc:1428] Setting key 9776d01e54d2c50ecc98050a0cd60064
I20260926 01:10:16.256857 5784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:16.252224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56782 (local address 127.0.98.62:33683)
0926 01:10:16.252443 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:16.252453 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:16.252486 (+ 33us) server_negotiation.cc:407] Connection header received
0926 01:10:16.252588 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:16.252594 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:16.252651 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:16.252776 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:16.252974 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:16.254219 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:16.254989 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:16.255409 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:16.255526 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:16.256201 (+ 675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:10:16.256205 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:10:16.256205 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:10:16.256378 (+ 173us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:10:16.256414 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:10:16.256424 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:10:16.256478 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:16.256559 (+ 81us) server_negotiation.cc:299] Negotiation successful
0926 01:10:16.256646 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":84,"threads_started":1}
I20260926 01:10:16.258232 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 01:10:16.261265 5780 raft_consensus.cc:504] T 00000000000000000000000000000000 P dc39be43155a4a5a900c7d3e9680e1c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc39be43155a4a5a900c7d3e9680e1c9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33683 } }
I20260926 01:10:16.261439 5780 leader_election.cc:302] T 00000000000000000000000000000000 P dc39be43155a4a5a900c7d3e9680e1c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dc39be43155a4a5a900c7d3e9680e1c9; no voters:
I20260926 01:10:16.261629 5780 leader_election.cc:288] T 00000000000000000000000000000000 P dc39be43155a4a5a900c7d3e9680e1c9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:10:16.261729 5785 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dc39be43155a4a5a900c7d3e9680e1c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:10:16.261929 5785 raft_consensus.cc:686] T 00000000000000000000000000000000 P dc39be43155a4a5a900c7d3e9680e1c9 [term 1 LEADER]: Becoming Leader. State: Replica: dc39be43155a4a5a900c7d3e9680e1c9, State: Running, Role: LEADER
I20260926 01:10:16.261936 5780 sys_catalog.cc:564] T 00000000000000000000000000000000 P dc39be43155a4a5a900c7d3e9680e1c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:10:16.262050 5785 consensus_queue.cc:237] T 00000000000000000000000000000000 P dc39be43155a4a5a900c7d3e9680e1c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc39be43155a4a5a900c7d3e9680e1c9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33683 } }
I20260926 01:10:16.263164 5786 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc39be43155a4a5a900c7d3e9680e1c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc39be43155a4a5a900c7d3e9680e1c9. Latest consensus state: current_term: 1 leader_uuid: "dc39be43155a4a5a900c7d3e9680e1c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc39be43155a4a5a900c7d3e9680e1c9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33683 } } }
I20260926 01:10:16.263266 5786 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc39be43155a4a5a900c7d3e9680e1c9 [sys.catalog]: This master's current role is: LEADER
I20260926 01:10:16.263370 5787 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc39be43155a4a5a900c7d3e9680e1c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dc39be43155a4a5a900c7d3e9680e1c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc39be43155a4a5a900c7d3e9680e1c9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33683 } } }
I20260926 01:10:16.263708 5787 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc39be43155a4a5a900c7d3e9680e1c9 [sys.catalog]: This master's current role is: LEADER
W20260926 01:10:16.264016 5800 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dc39be43155a4a5a900c7d3e9680e1c9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 01:10:16.264067 5800 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 01:10:16.264180 5801 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:10:16.264308 5801 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:10:16.266232 5801 catalog_manager.cc:1347] Generated new cluster ID: 2a5afe7a3dc24015bb635c91669c0399
I20260926 01:10:16.266284 5801 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 01:10:16.282132 5801 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:10:16.282759 5801 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:10:16.305114 5801 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dc39be43155a4a5a900c7d3e9680e1c9: Generated new TSK 0
I20260926 01:10:16.305326 5801 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 01:10:16.316411 5784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:16.312629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56790 (local address 127.0.98.62:33683)
0926 01:10:16.312676 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:16.312678 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:16.312736 (+ 58us) server_negotiation.cc:407] Connection header received
0926 01:10:16.312826 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:16.312826 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:16.312836 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:16.312896 (+ 60us) server_negotiation.cc:226] Negotiated authn=JWT
0926 01:10:16.313094 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:16.313564 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:16.314749 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:16.314958 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:16.315018 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:16.315213 (+ 195us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 01:10:16.316167 (+ 954us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42845/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0926 01:10:16.316312 (+ 145us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56790: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42845/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":21}
W20260926 01:10:16.316463 5809 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:10:16.312538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33683 (local address 127.0.0.1:56790)
0926 01:10:16.312676 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:16.312687 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:16.312756 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:16.312912 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:16.312913 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:16.312916 (+ 3us) client_negotiation.cc:189] Negotiated authn=JWT
0926 01:10:16.312976 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:16.312978 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:16.313646 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:16.313647 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:16.314649 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:16.314652 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:16.315030 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:16.315037 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:16.315100 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:16.315106 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 01:10:16.316278 (+ 1172us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42845/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0926 01:10:16.316374 (+ 96us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:33683: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42845/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":71,"threads_started":1}
W20260926 01:10:16.316493 5784 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56790: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42845/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
I20260926 01:10:16.316941 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 5713
E20260926 01:10:16.329437 5810 webserver.cc:533] Webserver: sslize failed:
E20260926 01:10:16.329536 5810 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-26T01:10:16Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (238 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-26T01:10:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:10:16Z Disabled control of system clock
I20260926 01:10:16.363754 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790384993574762-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790384993574762-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790384993574762-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41029
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:33545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41029
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790384993574762-392-0/testchainca.cert with env {}
W20260926 01:10:16.447907 5814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:10:16.448159 5814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:10:16.448185 5814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:10:16.449527 5814 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:10:16.449582 5814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:10:16.449594 5814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:10:16.449605 5814 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:10:16.449616 5814 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 01:10:16.449627 5814 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 01:10:16.449645 5814 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260926 01:10:16.451354 5814 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790384993574762-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790384993574762-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41029
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41029
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790384993574762-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5814
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790384993574762-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:10:16.451628 5814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:10:16.451884 5814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:10:16.456844 5820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:10:16.456849 5822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:10:16.456849 5819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:10:16.457118 5814 server_base.cc:1034] running on GCE node
I20260926 01:10:16.457440 5814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:10:16.457751 5814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:10:16.458889 5814 hybrid_clock.cc:648] HybridClock initialized: now 1790385016458873 us; error 49 us; skew 500 ppm
I20260926 01:10:16.460479 5814 webserver.cc:460] Webserver started at http://127.0.98.62:44037/ using document root <none> and password file <none>
I20260926 01:10:16.460716 5814 fs_manager.cc:359] Metadata directory not provided
I20260926 01:10:16.460762 5814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:10:16.460876 5814 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:10:16.462001 5814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790384993574762-392-0/minicluster-data/master-0/data/instance:
uuid: "ec1f236ff9fc467e8ed86684aef22b3c"
format_stamp: "Formatted at 2026-09-26 01:10:16 on dist-test-slave-7b22"
server_key: "f33b764b1f45a5211061eb8d21fdfe42"
server_key_iv: "4ad754dfade349c693cee6c2c2bb1044"
server_key_version: "encryptionkey@0"
I20260926 01:10:16.462328 5814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790384993574762-392-0/minicluster-data/master-0/wal/instance:
uuid: "ec1f236ff9fc467e8ed86684aef22b3c"
format_stamp: "Formatted at 2026-09-26 01:10:16 on dist-test-slave-7b22"
server_key: "f33b764b1f45a5211061eb8d21fdfe42"
server_key_iv: "4ad754dfade349c693cee6c2c2bb1044"
server_key_version: "encryptionkey@0"
I20260926 01:10:16.463656 5814 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 01:10:16.464529 5828 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:16.464731 5814 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:10:16.464831 5814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790384993574762-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790384993574762-392-0/minicluster-data/master-0/wal
uuid: "ec1f236ff9fc467e8ed86684aef22b3c"
format_stamp: "Formatted at 2026-09-26 01:10:16 on dist-test-slave-7b22"
server_key: "f33b764b1f45a5211061eb8d21fdfe42"
server_key_iv: "4ad754dfade349c693cee6c2c2bb1044"
server_key_version: "encryptionkey@0"
I20260926 01:10:16.464907 5814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790384993574762-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790384993574762-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790384993574762-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:10:16.483786 5814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:10:16.484122 5814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:10:16.484256 5814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:10:16.488409 5814 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41029
I20260926 01:10:16.488494 5880 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41029 every 8 connection(s)
I20260926 01:10:16.488811 5814 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
I20260926 01:10:16.489454 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 5814
I20260926 01:10:16.489555 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790384993574762-392-0/minicluster-data/master-0/wal/instance
I20260926 01:10:16.489753 392 external_mini_cluster.cc:1428] Setting key d9115c61356f8f0b3a4bc1a70bd7d468
I20260926 01:10:16.489780 5881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:10:16.493367 5881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec1f236ff9fc467e8ed86684aef22b3c: Bootstrap starting.
I20260926 01:10:16.494082 5881 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ec1f236ff9fc467e8ed86684aef22b3c: Neither blocks nor log segments found. Creating new log.
I20260926 01:10:16.494365 5881 log.cc:824] T 00000000000000000000000000000000 P ec1f236ff9fc467e8ed86684aef22b3c: Log is configured to *not* fsync() on all Append() calls
I20260926 01:10:16.495178 5881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec1f236ff9fc467e8ed86684aef22b3c: No bootstrap required, opened a new log
I20260926 01:10:16.496440 5884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:16.490324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58228 (local address 127.0.98.62:41029)
0926 01:10:16.490568 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:16.490577 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:16.490617 (+ 40us) server_negotiation.cc:407] Connection header received
0926 01:10:16.491370 (+ 753us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:16.491381 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:16.491475 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:16.491639 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:16.491965 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:16.493305 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:16.494225 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:16.494707 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:16.494830 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:16.495666 (+ 836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:10:16.495684 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:10:16.495684 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:10:16.495889 (+ 205us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:10:16.495934 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:10:16.495950 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:10:16.496023 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:16.496103 (+ 80us) server_negotiation.cc:299] Negotiation successful
0926 01:10:16.496190 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":107,"threads_started":1}
I20260926 01:10:16.497906 5881 raft_consensus.cc:348] T 00000000000000000000000000000000 P ec1f236ff9fc467e8ed86684aef22b3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec1f236ff9fc467e8ed86684aef22b3c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41029 } }
I20260926 01:10:16.498111 5881 raft_consensus.cc:374] T 00000000000000000000000000000000 P ec1f236ff9fc467e8ed86684aef22b3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:10:16.498145 5881 raft_consensus.cc:729] T 00000000000000000000000000000000 P ec1f236ff9fc467e8ed86684aef22b3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec1f236ff9fc467e8ed86684aef22b3c, State: Initialized, Role: FOLLOWER
I20260926 01:10:16.498287 5881 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec1f236ff9fc467e8ed86684aef22b3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec1f236ff9fc467e8ed86684aef22b3c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41029 } }
I20260926 01:10:16.498380 5881 raft_consensus.cc:388] T 00000000000000000000000000000000 P ec1f236ff9fc467e8ed86684aef22b3c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:10:16.498421 5881 raft_consensus.cc:482] T 00000000000000000000000000000000 P ec1f236ff9fc467e8ed86684aef22b3c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:10:16.498473 5881 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ec1f236ff9fc467e8ed86684aef22b3c [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:10:16.500329 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 01:10:16.502434 5881 raft_consensus.cc:504] T 00000000000000000000000000000000 P ec1f236ff9fc467e8ed86684aef22b3c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec1f236ff9fc467e8ed86684aef22b3c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41029 } }
I20260926 01:10:16.502683 5881 leader_election.cc:302] T 00000000000000000000000000000000 P ec1f236ff9fc467e8ed86684aef22b3c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ec1f236ff9fc467e8ed86684aef22b3c; no voters:
I20260926 01:10:16.502861 5881 leader_election.cc:288] T 00000000000000000000000000000000 P ec1f236ff9fc467e8ed86684aef22b3c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:10:16.502940 5886 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ec1f236ff9fc467e8ed86684aef22b3c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:10:16.503139 5881 sys_catalog.cc:564] T 00000000000000000000000000000000 P ec1f236ff9fc467e8ed86684aef22b3c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:10:16.503273 5886 raft_consensus.cc:686] T 00000000000000000000000000000000 P ec1f236ff9fc467e8ed86684aef22b3c [term 1 LEADER]: Becoming Leader. State: Replica: ec1f236ff9fc467e8ed86684aef22b3c, State: Running, Role: LEADER
I20260926 01:10:16.503389 5886 consensus_queue.cc:237] T 00000000000000000000000000000000 P ec1f236ff9fc467e8ed86684aef22b3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec1f236ff9fc467e8ed86684aef22b3c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41029 } }
I20260926 01:10:16.503810 5887 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec1f236ff9fc467e8ed86684aef22b3c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ec1f236ff9fc467e8ed86684aef22b3c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec1f236ff9fc467e8ed86684aef22b3c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41029 } } }
I20260926 01:10:16.503929 5887 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec1f236ff9fc467e8ed86684aef22b3c [sys.catalog]: This master's current role is: LEADER
I20260926 01:10:16.504101 5888 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec1f236ff9fc467e8ed86684aef22b3c [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec1f236ff9fc467e8ed86684aef22b3c. Latest consensus state: current_term: 1 leader_uuid: "ec1f236ff9fc467e8ed86684aef22b3c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec1f236ff9fc467e8ed86684aef22b3c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41029 } } }
I20260926 01:10:16.504174 5888 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec1f236ff9fc467e8ed86684aef22b3c [sys.catalog]: This master's current role is: LEADER
I20260926 01:10:16.504464 5895 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:10:16.505003 5895 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:10:16.507014 5895 catalog_manager.cc:1347] Generated new cluster ID: 6cd6cbdfca6d4cffb84b839ff28898f0
I20260926 01:10:16.507085 5895 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 01:10:16.512686 5895 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:10:16.513407 5895 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:10:16.523452 5895 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ec1f236ff9fc467e8ed86684aef22b3c: Generated new TSK 0
I20260926 01:10:16.523695 5895 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 01:10:16.594513 5884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:16.533684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58236 (local address 127.0.98.62:41029)
0926 01:10:16.533733 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:16.533735 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:16.533842 (+ 107us) server_negotiation.cc:407] Connection header received
0926 01:10:16.533921 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:16.533922 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:16.533936 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:16.534024 (+ 88us) server_negotiation.cc:226] Negotiated authn=JWT
0926 01:10:16.534193 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:16.534964 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:16.535775 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:16.536092 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:16.536169 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:16.536342 (+ 173us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 01:10:16.594187 (+ 57845us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0926 01:10:16.594428 (+ 241us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58236: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":26}
W20260926 01:10:16.594592 5884 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58236: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260926 01:10:16.594702 5910 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:10:16.533607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41029 (local address 127.0.0.1:58236)
0926 01:10:16.533749 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:16.533759 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:16.533848 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:16.534060 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:16.534062 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:16.534064 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0926 01:10:16.534121 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:16.534123 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:16.535065 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:16.535066 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:16.535690 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:16.535692 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:16.536173 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:16.536179 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:16.536227 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:16.536232 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 01:10:16.594391 (+ 58159us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0926 01:10:16.594593 (+ 202us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:41029: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":74,"threads_started":1}
I20260926 01:10:16.595088 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 5814
2026-09-26T01:10:16Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (256 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (95 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (93 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384993574762-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 01:10:16 dist-test-slave-7b22 krb5kdc[5922](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 01:10:16 dist-test-slave-7b22 krb5kdc[5922](info): set up 2 sockets
Sep 26 01:10:16 dist-test-slave-7b22 krb5kdc[5922](info): commencing operation
krb5kdc: starting...
W20260926 01:10:17.749876 392 mini_kdc.cc:121] Time spent starting KDC: real 0.949s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384993574762-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384993574762-392-0/krb5kdc/test-user.keytab.
Sep 26 01:10:17 dist-test-slave-7b22 krb5kdc[5922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385017, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T01:10:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:10:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384993574762-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384993574762-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 01:10:17.869968 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384993574762-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384993574762-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384993574762-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34949
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384993574762-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34949
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:10:17.953423 5935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:10:17.953672 5935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:10:17.953706 5935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:10:17.955001 5935 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:10:17.955051 5935 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:10:17.955070 5935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:10:17.955085 5935 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:10:17.955101 5935 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:10:17.956836 5935 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384993574762-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384993574762-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34949
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384993574762-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34949
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5935
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384993574762-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:10:17.957229 5935 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:10:17.957485 5935 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:10:17.963148 5940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:10:17.963181 5941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:10:17.963150 5943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:10:17.963218 5935 server_base.cc:1034] running on GCE node
I20260926 01:10:17.963655 5935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:10:17.963953 5935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:10:17.965143 5935 hybrid_clock.cc:648] HybridClock initialized: now 1790385017965105 us; error 48 us; skew 500 ppm
Sep 26 01:10:17 dist-test-slave-7b22 krb5kdc[5922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385017, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:10:17.967800 5935 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 01:10:17.968335 5935 webserver.cc:460] Webserver started at http://127.0.98.62:44009/ using document root <none> and password file <none>
I20260926 01:10:17.968557 5935 fs_manager.cc:359] Metadata directory not provided
I20260926 01:10:17.968608 5935 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:10:17.968746 5935 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:10:17.969888 5935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384993574762-392-0/minicluster-data/master-0/data/instance:
uuid: "41c40286ccd140bbb98bdea4d19715ec"
format_stamp: "Formatted at 2026-09-26 01:10:17 on dist-test-slave-7b22"
server_key: "a649445732e3083af299968eb5dc9c55"
server_key_iv: "ffafbd4d5285d428918a223c25073abe"
server_key_version: "encryptionkey@0"
I20260926 01:10:17.970238 5935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384993574762-392-0/minicluster-data/master-0/wal/instance:
uuid: "41c40286ccd140bbb98bdea4d19715ec"
format_stamp: "Formatted at 2026-09-26 01:10:17 on dist-test-slave-7b22"
server_key: "a649445732e3083af299968eb5dc9c55"
server_key_iv: "ffafbd4d5285d428918a223c25073abe"
server_key_version: "encryptionkey@0"
I20260926 01:10:17.971634 5935 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 01:10:17.972527 5950 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:17.972909 5935 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 01:10:17.973013 5935 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384993574762-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384993574762-392-0/minicluster-data/master-0/wal
uuid: "41c40286ccd140bbb98bdea4d19715ec"
format_stamp: "Formatted at 2026-09-26 01:10:17 on dist-test-slave-7b22"
server_key: "a649445732e3083af299968eb5dc9c55"
server_key_iv: "ffafbd4d5285d428918a223c25073abe"
server_key_version: "encryptionkey@0"
I20260926 01:10:17.973090 5935 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384993574762-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384993574762-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384993574762-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:10:17.987421 5935 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:10:17.989650 5935 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:10:17.989809 5935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:10:17.993700 5935 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34949
I20260926 01:10:17.994053 6002 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34949 every 8 connection(s)
I20260926 01:10:17.994105 5935 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
I20260926 01:10:17.994750 6003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:10:17.994926 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 5935
I20260926 01:10:17.995033 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384993574762-392-0/minicluster-data/master-0/wal/instance
I20260926 01:10:17.995249 392 external_mini_cluster.cc:1428] Setting key 8c636e7d18c92210d8b3bca49ff6b67f
I20260926 01:10:17.998031 6003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41c40286ccd140bbb98bdea4d19715ec: Bootstrap starting.
I20260926 01:10:17.998693 6003 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 41c40286ccd140bbb98bdea4d19715ec: Neither blocks nor log segments found. Creating new log.
I20260926 01:10:17.998986 6003 log.cc:824] T 00000000000000000000000000000000 P 41c40286ccd140bbb98bdea4d19715ec: Log is configured to *not* fsync() on all Append() calls
I20260926 01:10:17.999678 6003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41c40286ccd140bbb98bdea4d19715ec: No bootstrap required, opened a new log
I20260926 01:10:18.001650 6003 raft_consensus.cc:348] T 00000000000000000000000000000000 P 41c40286ccd140bbb98bdea4d19715ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41c40286ccd140bbb98bdea4d19715ec" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34949 } }
Sep 26 01:10:17 dist-test-slave-7b22 krb5kdc[5922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385017, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:10:18.001860 6003 raft_consensus.cc:374] T 00000000000000000000000000000000 P 41c40286ccd140bbb98bdea4d19715ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:10:18.001899 6003 raft_consensus.cc:729] T 00000000000000000000000000000000 P 41c40286ccd140bbb98bdea4d19715ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41c40286ccd140bbb98bdea4d19715ec, State: Initialized, Role: FOLLOWER
I20260926 01:10:18.002039 6003 consensus_queue.cc:260] T 00000000000000000000000000000000 P 41c40286ccd140bbb98bdea4d19715ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41c40286ccd140bbb98bdea4d19715ec" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34949 } }
I20260926 01:10:18.002126 6003 raft_consensus.cc:388] T 00000000000000000000000000000000 P 41c40286ccd140bbb98bdea4d19715ec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:10:18.002166 6003 raft_consensus.cc:482] T 00000000000000000000000000000000 P 41c40286ccd140bbb98bdea4d19715ec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:10:18.002213 6003 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 41c40286ccd140bbb98bdea4d19715ec [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:10:18.002863 6003 raft_consensus.cc:504] T 00000000000000000000000000000000 P 41c40286ccd140bbb98bdea4d19715ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41c40286ccd140bbb98bdea4d19715ec" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34949 } }
I20260926 01:10:18.002964 6003 leader_election.cc:302] T 00000000000000000000000000000000 P 41c40286ccd140bbb98bdea4d19715ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 41c40286ccd140bbb98bdea4d19715ec; no voters:
I20260926 01:10:18.003149 6003 leader_election.cc:288] T 00000000000000000000000000000000 P 41c40286ccd140bbb98bdea4d19715ec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:10:18.003401 6003 sys_catalog.cc:564] T 00000000000000000000000000000000 P 41c40286ccd140bbb98bdea4d19715ec [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:10:18.003561 6008 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 41c40286ccd140bbb98bdea4d19715ec [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:10:18.003718 6008 raft_consensus.cc:686] T 00000000000000000000000000000000 P 41c40286ccd140bbb98bdea4d19715ec [term 1 LEADER]: Becoming Leader. State: Replica: 41c40286ccd140bbb98bdea4d19715ec, State: Running, Role: LEADER
I20260926 01:10:18.003824 6008 consensus_queue.cc:237] T 00000000000000000000000000000000 P 41c40286ccd140bbb98bdea4d19715ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41c40286ccd140bbb98bdea4d19715ec" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34949 } }
I20260926 01:10:18.004633 6010 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41c40286ccd140bbb98bdea4d19715ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 41c40286ccd140bbb98bdea4d19715ec. Latest consensus state: current_term: 1 leader_uuid: "41c40286ccd140bbb98bdea4d19715ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41c40286ccd140bbb98bdea4d19715ec" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34949 } } }
I20260926 01:10:18.004838 6009 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41c40286ccd140bbb98bdea4d19715ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "41c40286ccd140bbb98bdea4d19715ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41c40286ccd140bbb98bdea4d19715ec" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34949 } } }
I20260926 01:10:18.004876 6010 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41c40286ccd140bbb98bdea4d19715ec [sys.catalog]: This master's current role is: LEADER
I20260926 01:10:18.004887 6009 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41c40286ccd140bbb98bdea4d19715ec [sys.catalog]: This master's current role is: LEADER
W20260926 01:10:18.005017 6022 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 41c40286ccd140bbb98bdea4d19715ec: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 01:10:18.005079 6022 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 01:10:18.005203 6024 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:10:18.005398 6024 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:10:18.006444 6006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:17.995779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39908 (local address 127.0.98.62:34949)
0926 01:10:17.995947 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:17.995956 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:17.995981 (+ 25us) server_negotiation.cc:407] Connection header received
0926 01:10:17.996418 (+ 437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:17.996425 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:17.996474 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:17.996591 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:17.997350 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:17.998341 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:17.999321 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:17.999811 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:17.999909 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:18.002421 (+ 2512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:10:18.002426 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:10:18.002426 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:10:18.002437 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:10:18.004714 (+ 2277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:18.005174 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:18.005178 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:18.005178 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:18.005281 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:18.005477 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:18.005478 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:18.005478 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:18.005728 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:10:18.005902 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:18.006077 (+ 175us) server_negotiation.cc:299] Negotiation successful
0926 01:10:18.006223 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":71,"threads_started":1}
I20260926 01:10:18.007117 6024 catalog_manager.cc:1347] Generated new cluster ID: 3b5ecb37aa404d55a77ebbd20504f5ba
I20260926 01:10:18.007158 6024 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 01:10:18.015437 6024 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:10:18.016001 6024 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:10:18.021082 6024 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 41c40286ccd140bbb98bdea4d19715ec: Generated new TSK 0
I20260926 01:10:18.021292 6024 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384993574762-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384993574762-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 01:10:18.049294 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384993574762-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384993574762-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384993574762-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384993574762-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34949
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384993574762-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:10:18.133507 6029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:10:18.133865 6029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:10:18.133909 6029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:10:18.135474 6029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:10:18.135540 6029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:10:18.135571 6029 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 01:10:18.137247 6029 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384993574762-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384993574762-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384993574762-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384993574762-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34949
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6029
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384993574762-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:10:18.137502 6029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:10:18.137750 6029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:10:18.142794 6035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:10:18.142925 6029 server_base.cc:1034] running on GCE node
W20260926 01:10:18.142798 6034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:10:18.143013 6037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:10:18.143275 6029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:10:18.143519 6029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:10:18.144694 6029 hybrid_clock.cc:648] HybridClock initialized: now 1790385018144661 us; error 47 us; skew 500 ppm
Sep 26 01:10:18 dist-test-slave-7b22 krb5kdc[5922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385018, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:10:18.147455 6029 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 01:10:18.147955 6029 webserver.cc:460] Webserver started at http://127.0.98.1:37635/ using document root <none> and password file <none>
I20260926 01:10:18.148160 6029 fs_manager.cc:359] Metadata directory not provided
I20260926 01:10:18.148204 6029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:10:18.148325 6029 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:10:18.149416 6029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384993574762-392-0/minicluster-data/ts-0/data/instance:
uuid: "901785d58ab742bfac3b83b6a7ada336"
format_stamp: "Formatted at 2026-09-26 01:10:18 on dist-test-slave-7b22"
server_key: "d3df2524ac1693d374dc45ef11376fb2"
server_key_iv: "abd90f75691455f7c76fa4994a2c146b"
server_key_version: "encryptionkey@0"
I20260926 01:10:18.149758 6029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384993574762-392-0/minicluster-data/ts-0/wal/instance:
uuid: "901785d58ab742bfac3b83b6a7ada336"
format_stamp: "Formatted at 2026-09-26 01:10:18 on dist-test-slave-7b22"
server_key: "d3df2524ac1693d374dc45ef11376fb2"
server_key_iv: "abd90f75691455f7c76fa4994a2c146b"
server_key_version: "encryptionkey@0"
I20260926 01:10:18.151199 6029 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 01:10:18.152215 6044 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:18.152458 6029 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 01:10:18.152540 6029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384993574762-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384993574762-392-0/minicluster-data/ts-0/wal
uuid: "901785d58ab742bfac3b83b6a7ada336"
format_stamp: "Formatted at 2026-09-26 01:10:18 on dist-test-slave-7b22"
server_key: "d3df2524ac1693d374dc45ef11376fb2"
server_key_iv: "abd90f75691455f7c76fa4994a2c146b"
server_key_version: "encryptionkey@0"
I20260926 01:10:18.152623 6029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384993574762-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384993574762-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384993574762-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:10:18.160395 6029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:10:18.162585 6029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:10:18.162708 6029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:10:18.162953 6029 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:10:18.163257 6029 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:10:18.163291 6029 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:18.163321 6029 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:10:18.163344 6029 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:18.169011 6029 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38869
I20260926 01:10:18.169067 6157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38869 every 8 connection(s)
I20260926 01:10:18.169365 6029 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384993574762-392-0/minicluster-data/ts-0/data/info.pb
I20260926 01:10:18.174223 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 6029
I20260926 01:10:18.174350 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384993574762-392-0/minicluster-data/ts-0/wal/instance
I20260926 01:10:18.174535 392 external_mini_cluster.cc:1428] Setting key f9f50f0e863cb9f95ef66fc53b1d4598
Sep 26 01:10:18 dist-test-slave-7b22 krb5kdc[5922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385018, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:10:18.179612 6006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:18.170083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48995 (local address 127.0.98.62:34949)
0926 01:10:18.170159 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:18.170161 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:18.170311 (+ 150us) server_negotiation.cc:407] Connection header received
0926 01:10:18.171257 (+ 946us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:18.171259 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:18.171269 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:18.171317 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:18.171963 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:18.172487 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:18.173532 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:18.173750 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:18.173810 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:18.176476 (+ 2666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:10:18.176478 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:10:18.176479 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:10:18.176486 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:10:18.178037 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:18.178531 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:18.178532 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:18.178532 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:18.178587 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:18.178893 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:18.178894 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:18.178894 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:18.179040 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:10:18.179111 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:18.179454 (+ 343us) server_negotiation.cc:299] Negotiation successful
0926 01:10:18.179533 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 01:10:18.179872 6160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:18.170052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34949 (local address 127.0.98.1:48995)
0926 01:10:18.170191 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:18.170231 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:18.171156 (+ 925us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:18.171381 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:18.171385 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:18.171657 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:10:18.171861 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:18.171866 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:18.172582 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:18.172583 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:18.173438 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:18.173441 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:18.173867 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:18.173869 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:18.174005 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:18.174605 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:10:18.174616 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:10:18.176366 (+ 1750us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:10:18.178129 (+ 1763us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:10:18.178138 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:10:18.178141 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:10:18.178438 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:10:18.178648 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:10:18.178649 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:10:18.178650 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:10:18.178807 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:10:18.179108 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:10:18.179114 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:10:18.179298 (+ 184us) client_negotiation.cc:769] Sending connection context
0926 01:10:18.179426 (+ 128us) client_negotiation.cc:240] Negotiation successful
0926 01:10:18.179615 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":73,"threads_started":1}
I20260926 01:10:18.180464 6158 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34949
I20260926 01:10:18.180574 6158 heartbeater.cc:464] Registering TS with master...
I20260926 01:10:18.180764 6158 heartbeater.cc:511] Master 127.0.98.62:34949 requested a full tablet report, sending...
I20260926 01:10:18.181336 5967 ts_manager.cc:194] Registered new tserver with Master: 901785d58ab742bfac3b83b6a7ada336 (127.0.98.1:38869)
I20260926 01:10:18.182309 5967 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:48995
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384993574762-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384993574762-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 01:10:18.201893 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384993574762-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384993574762-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384993574762-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384993574762-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34949
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384993574762-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:10:18.291780 6163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:10:18.292024 6163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:10:18.292052 6163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:10:18.293438 6163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:10:18.293493 6163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:10:18.293525 6163 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 01:10:18.295291 6163 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384993574762-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384993574762-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384993574762-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384993574762-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34949
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6163
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384993574762-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:10:18.295569 6163 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:10:18.295792 6163 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:10:18.300770 6171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:10:18.300925 6169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:10:18.300783 6168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:10:18.301043 6163 server_base.cc:1034] running on GCE node
I20260926 01:10:18.301326 6163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:10:18.301568 6163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:10:18.302729 6163 hybrid_clock.cc:648] HybridClock initialized: now 1790385018302711 us; error 32 us; skew 500 ppm
Sep 26 01:10:18 dist-test-slave-7b22 krb5kdc[5922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385018, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:10:18.305868 6163 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 01:10:18.306366 6163 webserver.cc:460] Webserver started at http://127.0.98.2:44205/ using document root <none> and password file <none>
I20260926 01:10:18.306607 6163 fs_manager.cc:359] Metadata directory not provided
I20260926 01:10:18.306658 6163 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:10:18.306795 6163 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:10:18.307883 6163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384993574762-392-0/minicluster-data/ts-1/data/instance:
uuid: "088ae4a51c12440f8fc5ed146c3c13ae"
format_stamp: "Formatted at 2026-09-26 01:10:18 on dist-test-slave-7b22"
server_key: "8216a6f92019b2248deaab78f200dda2"
server_key_iv: "a5739a45a0fbfbc8da01ebf88a1660c5"
server_key_version: "encryptionkey@0"
I20260926 01:10:18.308225 6163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384993574762-392-0/minicluster-data/ts-1/wal/instance:
uuid: "088ae4a51c12440f8fc5ed146c3c13ae"
format_stamp: "Formatted at 2026-09-26 01:10:18 on dist-test-slave-7b22"
server_key: "8216a6f92019b2248deaab78f200dda2"
server_key_iv: "a5739a45a0fbfbc8da01ebf88a1660c5"
server_key_version: "encryptionkey@0"
I20260926 01:10:18.309562 6163 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 01:10:18.310426 6178 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:18.310607 6163 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 01:10:18.310740 6163 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384993574762-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384993574762-392-0/minicluster-data/ts-1/wal
uuid: "088ae4a51c12440f8fc5ed146c3c13ae"
format_stamp: "Formatted at 2026-09-26 01:10:18 on dist-test-slave-7b22"
server_key: "8216a6f92019b2248deaab78f200dda2"
server_key_iv: "a5739a45a0fbfbc8da01ebf88a1660c5"
server_key_version: "encryptionkey@0"
I20260926 01:10:18.310815 6163 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384993574762-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384993574762-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384993574762-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:10:18.324386 6163 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:10:18.326588 6163 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:10:18.326741 6163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:10:18.326961 6163 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:10:18.327270 6163 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:10:18.327303 6163 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:18.327332 6163 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:10:18.327354 6163 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:18.332813 6163 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42633
I20260926 01:10:18.332885 6291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42633 every 8 connection(s)
I20260926 01:10:18.333179 6163 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384993574762-392-0/minicluster-data/ts-1/data/info.pb
I20260926 01:10:18.337054 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 6163
I20260926 01:10:18.337144 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384993574762-392-0/minicluster-data/ts-1/wal/instance
I20260926 01:10:18.337327 392 external_mini_cluster.cc:1428] Setting key a83c8cd30a33980ea7c08152d82af788
Sep 26 01:10:18 dist-test-slave-7b22 krb5kdc[5922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385018, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:10:18.343094 6006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:18.333872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53777 (local address 127.0.98.62:34949)
0926 01:10:18.333936 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:18.333938 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:18.334176 (+ 238us) server_negotiation.cc:407] Connection header received
0926 01:10:18.335166 (+ 990us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:18.335167 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:18.335178 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:18.335247 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:18.335850 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:18.336372 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:18.337494 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:18.337728 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:18.337811 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:18.340237 (+ 2426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:10:18.340239 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:10:18.340239 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:10:18.340247 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:10:18.341499 (+ 1252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:18.342046 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:18.342047 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:18.342047 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:18.342110 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:18.342388 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:18.342389 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:18.342389 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:18.342499 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:10:18.342545 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:18.342931 (+ 386us) server_negotiation.cc:299] Negotiation successful
0926 01:10:18.343022 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 01:10:18.343297 6294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:18.333873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34949 (local address 127.0.98.2:53777)
0926 01:10:18.334043 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:18.334079 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:18.335047 (+ 968us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:18.335306 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:18.335309 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:18.335565 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:10:18.335767 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:18.335772 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:18.336458 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:18.336459 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:18.337403 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:18.337406 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:18.337843 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:18.337844 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:18.337992 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:18.338496 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:10:18.338505 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:10:18.340114 (+ 1609us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:10:18.341575 (+ 1461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:10:18.341583 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:10:18.341586 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:10:18.341949 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:10:18.342178 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:10:18.342179 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:10:18.342179 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:10:18.342326 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:10:18.342552 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:10:18.342558 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:10:18.342746 (+ 188us) client_negotiation.cc:769] Sending connection context
0926 01:10:18.342873 (+ 127us) client_negotiation.cc:240] Negotiation successful
0926 01:10:18.343056 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":97,"threads_started":1}
I20260926 01:10:18.343755 6292 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34949
I20260926 01:10:18.343851 6292 heartbeater.cc:464] Registering TS with master...
I20260926 01:10:18.344034 6292 heartbeater.cc:511] Master 127.0.98.62:34949 requested a full tablet report, sending...
I20260926 01:10:18.344419 5967 ts_manager.cc:194] Registered new tserver with Master: 088ae4a51c12440f8fc5ed146c3c13ae (127.0.98.2:42633)
I20260926 01:10:18.344803 5967 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:53777
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384993574762-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384993574762-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 01:10:18.364883 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384993574762-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384993574762-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384993574762-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384993574762-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34949
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384993574762-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:10:18.446975 6297 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:10:18.447203 6297 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:10:18.447230 6297 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:10:18.448407 6297 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:10:18.448446 6297 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:10:18.448472 6297 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 01:10:18.450027 6297 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384993574762-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384993574762-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384993574762-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384993574762-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34949
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6297
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384993574762-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:10:18.450237 6297 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:10:18.450424 6297 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:10:18.455364 6303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:10:18.455507 6305 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:10:18.455364 6302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:10:18.455608 6297 server_base.cc:1034] running on GCE node
I20260926 01:10:18.455945 6297 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:10:18.456192 6297 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:10:18.457361 6297 hybrid_clock.cc:648] HybridClock initialized: now 1790385018457347 us; error 30 us; skew 500 ppm
Sep 26 01:10:18 dist-test-slave-7b22 krb5kdc[5922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385018, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:10:18.459933 6297 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 01:10:18.460408 6297 webserver.cc:460] Webserver started at http://127.0.98.3:46687/ using document root <none> and password file <none>
I20260926 01:10:18.460691 6297 fs_manager.cc:359] Metadata directory not provided
I20260926 01:10:18.460772 6297 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:10:18.460882 6297 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:10:18.462124 6297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384993574762-392-0/minicluster-data/ts-2/data/instance:
uuid: "b9b4ea5f5fb94744a3566a7531ada08e"
format_stamp: "Formatted at 2026-09-26 01:10:18 on dist-test-slave-7b22"
server_key: "e26a961d3f9d76e716a78c72869f8439"
server_key_iv: "7cdb75c54ff57906edb5956789bcf1c1"
server_key_version: "encryptionkey@0"
I20260926 01:10:18.462435 6297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384993574762-392-0/minicluster-data/ts-2/wal/instance:
uuid: "b9b4ea5f5fb94744a3566a7531ada08e"
format_stamp: "Formatted at 2026-09-26 01:10:18 on dist-test-slave-7b22"
server_key: "e26a961d3f9d76e716a78c72869f8439"
server_key_iv: "7cdb75c54ff57906edb5956789bcf1c1"
server_key_version: "encryptionkey@0"
I20260926 01:10:18.463722 6297 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 01:10:18.464609 6312 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:18.464854 6297 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:10:18.464941 6297 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384993574762-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384993574762-392-0/minicluster-data/ts-2/wal
uuid: "b9b4ea5f5fb94744a3566a7531ada08e"
format_stamp: "Formatted at 2026-09-26 01:10:18 on dist-test-slave-7b22"
server_key: "e26a961d3f9d76e716a78c72869f8439"
server_key_iv: "7cdb75c54ff57906edb5956789bcf1c1"
server_key_version: "encryptionkey@0"
I20260926 01:10:18.465021 6297 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384993574762-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384993574762-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384993574762-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:10:18.478608 6297 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:10:18.480751 6297 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:10:18.480930 6297 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:10:18.481175 6297 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:10:18.481496 6297 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:10:18.481528 6297 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:18.481552 6297 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:10:18.481575 6297 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:18.487452 6297 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39933
I20260926 01:10:18.487500 6425 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39933 every 8 connection(s)
I20260926 01:10:18.487830 6297 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384993574762-392-0/minicluster-data/ts-2/data/info.pb
I20260926 01:10:18.490016 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 6297
I20260926 01:10:18.490118 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384993574762-392-0/minicluster-data/ts-2/wal/instance
I20260926 01:10:18.490293 392 external_mini_cluster.cc:1428] Setting key c840bc3715b75ccd3c8da658acb5ae13
Sep 26 01:10:18 dist-test-slave-7b22 krb5kdc[5922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385018, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:10:18.497740 6006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:18.488606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50359 (local address 127.0.98.62:34949)
0926 01:10:18.488669 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:18.488671 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:18.488866 (+ 195us) server_negotiation.cc:407] Connection header received
0926 01:10:18.489805 (+ 939us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:18.489806 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:18.489851 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:18.489926 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:18.490667 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:18.491321 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:18.492334 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:18.492536 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:18.492598 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:18.494782 (+ 2184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:10:18.494784 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:10:18.494784 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:10:18.494791 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:10:18.496105 (+ 1314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:18.496637 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:18.496638 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:18.496638 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:18.496709 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:18.496996 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:18.496997 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:18.496997 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:18.497136 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:10:18.497215 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:18.497559 (+ 344us) server_negotiation.cc:299] Negotiation successful
0926 01:10:18.497646 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 01:10:18.497960 6428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:18.488566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34949 (local address 127.0.98.3:50359)
0926 01:10:18.488734 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:18.488777 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:18.489695 (+ 918us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:18.490002 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:18.490005 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:18.490306 (+ 301us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:10:18.490579 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:18.490586 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:18.491405 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:18.491406 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:18.492227 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:18.492229 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:18.492612 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:18.492613 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:18.492720 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:18.493183 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:10:18.493191 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:10:18.494687 (+ 1496us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:10:18.496180 (+ 1493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:10:18.496190 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:10:18.496193 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:10:18.496540 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:10:18.496788 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:10:18.496788 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:10:18.496789 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:10:18.496935 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:10:18.497207 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:10:18.497215 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:10:18.497412 (+ 197us) client_negotiation.cc:769] Sending connection context
0926 01:10:18.497525 (+ 113us) client_negotiation.cc:240] Negotiation successful
0926 01:10:18.497709 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":85,"threads_started":1}
I20260926 01:10:18.499013 6426 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34949
I20260926 01:10:18.499135 6426 heartbeater.cc:464] Registering TS with master...
I20260926 01:10:18.499337 6426 heartbeater.cc:511] Master 127.0.98.62:34949 requested a full tablet report, sending...
I20260926 01:10:18.499750 5967 ts_manager.cc:194] Registered new tserver with Master: b9b4ea5f5fb94744a3566a7531ada08e (127.0.98.3:39933)
I20260926 01:10:18.500255 5967 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:50359
I20260926 01:10:18.501964 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384993574762-392-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384993574762-392-0/krb5kdc/test-admin.keytab.
Sep 26 01:10:18 dist-test-slave-7b22 krb5kdc[5922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385018, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:10:18.518357 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260926 01:10:18.520262 6433 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:10:18.519414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57826 (local address 127.0.98.1:38869)
0926 01:10:18.519626 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:18.519631 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:18.519641 (+ 10us) server_negotiation.cc:407] Connection header received
0926 01:10:18.519670 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:18.519672 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:18.519696 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:18.519766 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:18.520179 (+ 413us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57826: BlockingRecv error: recv got EOF from 127.0.0.1:57826 (error 108)
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":125,"threads_started":1}
W20260926 01:10:18.520407 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384993574762-392-0/krb5kdc/krb5cc' of type FILE
Sep 26 01:10:18 dist-test-slave-7b22 krb5kdc[5922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385018, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:10:18.521631 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 01:10:18 dist-test-slave-7b22 krb5kdc[5922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385018, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:10:18.522677 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 01:10:18 dist-test-slave-7b22 krb5kdc[5922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385018, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 01:10:18.530586 6433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:18.523322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57842 (local address 127.0.98.1:38869)
0926 01:10:18.523370 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:18.523372 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:18.523383 (+ 11us) server_negotiation.cc:407] Connection header received
0926 01:10:18.523429 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:18.523429 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:18.523440 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:18.523487 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:18.523964 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:18.524728 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:18.525483 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:18.525893 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:18.525975 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:18.528131 (+ 2156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:10:18.528142 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:10:18.528142 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:10:18.528150 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:10:18.529512 (+ 1362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:18.529851 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:18.529854 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:18.529855 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:18.529915 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:18.530064 (+ 149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:18.530064 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:18.530065 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:18.530200 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:10:18.530317 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:18.530425 (+ 108us) server_negotiation.cc:299] Negotiation successful
0926 01:10:18.530490 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
W20260926 01:10:18.532263 6433 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:10:18.531805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57858 (local address 127.0.98.1:38869)
0926 01:10:18.531868 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:18.531870 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:18.531879 (+ 9us) server_negotiation.cc:407] Connection header received
0926 01:10:18.531925 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:18.531926 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:18.531935 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:18.531975 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:18.532197 (+ 222us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57858: BlockingRecv error: recv got EOF from 127.0.0.1:57858 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260926 01:10:18.532317 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384993574762-392-0/krb5kdc/krb5cc' of type FILE
Sep 26 01:10:18 dist-test-slave-7b22 krb5kdc[5922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385018, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:10:18.533262 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 01:10:18 dist-test-slave-7b22 krb5kdc[5922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385018, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:10:18.534246 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 01:10:18 dist-test-slave-7b22 krb5kdc[5922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385018, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 01:10:18.541278 6433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:18.534870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57864 (local address 127.0.98.1:38869)
0926 01:10:18.534949 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:18.534951 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:18.534960 (+ 9us) server_negotiation.cc:407] Connection header received
0926 01:10:18.534980 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:18.534981 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:18.534991 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:18.535052 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:18.535419 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:18.535929 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:18.536618 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:18.536850 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:18.536907 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:18.538917 (+ 2010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:10:18.538919 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:10:18.538919 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:10:18.538925 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:10:18.540221 (+ 1296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:18.540540 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:18.540541 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:18.540541 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:18.540589 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:18.540765 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:18.540766 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:18.540766 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:18.540885 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:10:18.540968 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:18.541115 (+ 147us) server_negotiation.cc:299] Negotiation successful
0926 01:10:18.541183 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260926 01:10:18.542830 6433 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:10:18.542358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57870 (local address 127.0.98.1:38869)
0926 01:10:18.542397 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:18.542398 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:18.542407 (+ 9us) server_negotiation.cc:407] Connection header received
0926 01:10:18.542462 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:18.542462 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:18.542470 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:18.542505 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:18.542780 (+ 275us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57870: BlockingRecv error: recv got EOF from 127.0.0.1:57870 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260926 01:10:18.542956 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384993574762-392-0/krb5kdc/krb5cc' of type FILE
Sep 26 01:10:18 dist-test-slave-7b22 krb5kdc[5922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385018, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:10:18.543883 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 01:10:18 dist-test-slave-7b22 krb5kdc[5922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385018, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:10:18.544858 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 01:10:18 dist-test-slave-7b22 krb5kdc[5922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385018, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 01:10:18.552126 6433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:18.545420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57884 (local address 127.0.98.1:38869)
0926 01:10:18.545472 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:18.545474 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:18.545580 (+ 106us) server_negotiation.cc:407] Connection header received
0926 01:10:18.545612 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:18.545613 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:18.545623 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:18.545664 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:18.546087 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:18.546576 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:18.547269 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:18.547461 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:18.547522 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:18.549570 (+ 2048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:10:18.549571 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:10:18.549571 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:10:18.549578 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:10:18.550969 (+ 1391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:18.551380 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:18.551382 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:18.551382 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:18.551449 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:18.551645 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:18.551646 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:18.551646 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:18.551770 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:10:18.551854 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:18.551968 (+ 114us) server_negotiation.cc:299] Negotiation successful
0926 01:10:18.552032 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 01:10:18.552867 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 6029
I20260926 01:10:18.557000 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 6163
I20260926 01:10:18.561286 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 6297
I20260926 01:10:18.564998 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 5935
2026-09-26T01:10:18Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1783 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 01:10:18 dist-test-slave-7b22 krb5kdc[6446](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 01:10:18 dist-test-slave-7b22 krb5kdc[6446](info): set up 2 sockets
Sep 26 01:10:18 dist-test-slave-7b22 krb5kdc[6446](info): commencing operation
krb5kdc: starting...
W20260926 01:10:19.529969 392 mini_kdc.cc:121] Time spent starting KDC: real 0.945s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/krb5kdc/test-user.keytab.
Sep 26 01:10:19 dist-test-slave-7b22 krb5kdc[6446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385019, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T01:10:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:10:19Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260926 01:10:19.648550 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45453
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45453
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:10:19.728721 6459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:10:19.728986 6459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:10:19.729019 6459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:10:19.730386 6459 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:10:19.730449 6459 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:10:19.730468 6459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:10:19.730484 6459 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:10:19.730499 6459 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:10:19.732643 6459 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45453
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45453
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6459
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:10:19.732961 6459 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:10:19.733187 6459 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:10:19.738339 6464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:10:19.738343 6467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:10:19.738343 6465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:10:19.738391 6459 server_base.cc:1034] running on GCE node
I20260926 01:10:19.738899 6459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:10:19.739168 6459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:10:19.740327 6459 hybrid_clock.cc:648] HybridClock initialized: now 1790385019740310 us; error 34 us; skew 500 ppm
Sep 26 01:10:19 dist-test-slave-7b22 krb5kdc[6446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385019, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:10:19.743105 6459 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260926 01:10:19.743578 6459 webserver.cc:460] Webserver started at http://127.0.98.62:33679/ using document root <none> and password file <none>
I20260926 01:10:19.743798 6459 fs_manager.cc:359] Metadata directory not provided
I20260926 01:10:19.743844 6459 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:10:19.743970 6459 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:10:19.745057 6459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/master-0/data/instance:
uuid: "5afbc257ec7449b189377b1ae0cbebd5"
format_stamp: "Formatted at 2026-09-26 01:10:19 on dist-test-slave-7b22"
server_key: "fbc8696e10f0b5d5878445a9eb971f83"
server_key_iv: "e89f0fcf9fa90b9e0f4f8c6de9ded8a9"
server_key_version: "encryptionkey@0"
I20260926 01:10:19.745406 6459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/master-0/wal/instance:
uuid: "5afbc257ec7449b189377b1ae0cbebd5"
format_stamp: "Formatted at 2026-09-26 01:10:19 on dist-test-slave-7b22"
server_key: "fbc8696e10f0b5d5878445a9eb971f83"
server_key_iv: "e89f0fcf9fa90b9e0f4f8c6de9ded8a9"
server_key_version: "encryptionkey@0"
I20260926 01:10:19.746831 6459 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 01:10:19.747659 6474 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:19.747989 6459 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 01:10:19.748092 6459 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/master-0/wal
uuid: "5afbc257ec7449b189377b1ae0cbebd5"
format_stamp: "Formatted at 2026-09-26 01:10:19 on dist-test-slave-7b22"
server_key: "fbc8696e10f0b5d5878445a9eb971f83"
server_key_iv: "e89f0fcf9fa90b9e0f4f8c6de9ded8a9"
server_key_version: "encryptionkey@0"
I20260926 01:10:19.748164 6459 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:10:19.758937 6459 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:10:19.761119 6459 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:10:19.761269 6459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:10:19.765008 6459 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45453
I20260926 01:10:19.765060 6526 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45453 every 8 connection(s)
I20260926 01:10:19.765372 6459 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
I20260926 01:10:19.766419 6527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:10:19.768721 6527 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5afbc257ec7449b189377b1ae0cbebd5: Bootstrap starting.
I20260926 01:10:19.769299 6527 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5afbc257ec7449b189377b1ae0cbebd5: Neither blocks nor log segments found. Creating new log.
I20260926 01:10:19.769558 6527 log.cc:824] T 00000000000000000000000000000000 P 5afbc257ec7449b189377b1ae0cbebd5: Log is configured to *not* fsync() on all Append() calls
I20260926 01:10:19.770295 6527 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5afbc257ec7449b189377b1ae0cbebd5: No bootstrap required, opened a new log
I20260926 01:10:19.771919 6527 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5afbc257ec7449b189377b1ae0cbebd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5afbc257ec7449b189377b1ae0cbebd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45453 } }
I20260926 01:10:19.772117 6527 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5afbc257ec7449b189377b1ae0cbebd5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:10:19.772148 6527 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5afbc257ec7449b189377b1ae0cbebd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5afbc257ec7449b189377b1ae0cbebd5, State: Initialized, Role: FOLLOWER
I20260926 01:10:19.772303 6527 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5afbc257ec7449b189377b1ae0cbebd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5afbc257ec7449b189377b1ae0cbebd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45453 } }
I20260926 01:10:19.772368 6527 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5afbc257ec7449b189377b1ae0cbebd5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:10:19.772403 6527 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5afbc257ec7449b189377b1ae0cbebd5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:10:19.772447 6527 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5afbc257ec7449b189377b1ae0cbebd5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:10:19.773294 6527 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5afbc257ec7449b189377b1ae0cbebd5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5afbc257ec7449b189377b1ae0cbebd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45453 } }
I20260926 01:10:19.773396 6527 leader_election.cc:302] T 00000000000000000000000000000000 P 5afbc257ec7449b189377b1ae0cbebd5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5afbc257ec7449b189377b1ae0cbebd5; no voters:
I20260926 01:10:19.773582 6527 leader_election.cc:288] T 00000000000000000000000000000000 P 5afbc257ec7449b189377b1ae0cbebd5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:10:19.773653 6530 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5afbc257ec7449b189377b1ae0cbebd5 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:10:19.773806 6530 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5afbc257ec7449b189377b1ae0cbebd5 [term 1 LEADER]: Becoming Leader. State: Replica: 5afbc257ec7449b189377b1ae0cbebd5, State: Running, Role: LEADER
I20260926 01:10:19.773873 6527 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5afbc257ec7449b189377b1ae0cbebd5 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:10:19.773890 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 6459
I20260926 01:10:19.773977 6530 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5afbc257ec7449b189377b1ae0cbebd5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5afbc257ec7449b189377b1ae0cbebd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45453 } }
I20260926 01:10:19.773993 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/master-0/wal/instance
I20260926 01:10:19.774200 392 external_mini_cluster.cc:1428] Setting key d1e243443ada9fffadae6f83c1bd35a9
I20260926 01:10:19.774312 6530 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5afbc257ec7449b189377b1ae0cbebd5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5afbc257ec7449b189377b1ae0cbebd5. Latest consensus state: current_term: 1 leader_uuid: "5afbc257ec7449b189377b1ae0cbebd5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5afbc257ec7449b189377b1ae0cbebd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45453 } } }
I20260926 01:10:19.774371 6530 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5afbc257ec7449b189377b1ae0cbebd5 [sys.catalog]: This master's current role is: LEADER
I20260926 01:10:19.774513 6531 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5afbc257ec7449b189377b1ae0cbebd5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5afbc257ec7449b189377b1ae0cbebd5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5afbc257ec7449b189377b1ae0cbebd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45453 } } }
I20260926 01:10:19.774580 6531 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5afbc257ec7449b189377b1ae0cbebd5 [sys.catalog]: This master's current role is: LEADER
I20260926 01:10:19.774701 6539 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:10:19.775244 6539 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:10:19.777017 6539 catalog_manager.cc:1347] Generated new cluster ID: d16d612222b24a05bd8871dd9f254e18
I20260926 01:10:19.777072 6539 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 01:10:19 dist-test-slave-7b22 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385019, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 01:10:19.784343 6544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:19.774854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44364 (local address 127.0.98.62:45453)
0926 01:10:19.774993 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:19.775003 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:19.775117 (+ 114us) server_negotiation.cc:407] Connection header received
0926 01:10:19.775326 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:19.775334 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:19.775409 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:19.775546 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:19.776374 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:19.777480 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:19.778287 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:19.778718 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:19.778818 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:19.781027 (+ 2209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:10:19.781032 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:10:19.781032 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:10:19.781045 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:10:19.782636 (+ 1591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:19.782989 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:19.782992 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:19.782993 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:19.783098 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:19.783274 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:19.783275 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:19.783275 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:19.783653 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:10:19.783893 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:19.783977 (+ 84us) server_negotiation.cc:299] Negotiation successful
0926 01:10:19.784126 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81,"thread_start_us":62,"threads_started":1}
I20260926 01:10:19.794410 6539 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:10:19.795063 6539 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:10:19.801239 6539 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5afbc257ec7449b189377b1ae0cbebd5: Generated new TSK 0
I20260926 01:10:19.801486 6539 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260926 01:10:19.833496 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45453
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:10:19.916025 6553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:10:19.916241 6553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:10:19.916265 6553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:10:19.917431 6553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:10:19.917471 6553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:10:19.917496 6553 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 01:10:19.919121 6553 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45453
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6553
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:10:19.919330 6553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:10:19.919523 6553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:10:19.925066 6559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:10:19.925160 6561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:10:19.925176 6558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:10:19.925431 6553 server_base.cc:1034] running on GCE node
I20260926 01:10:19.925629 6553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:10:19.925899 6553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:10:19.927050 6553 hybrid_clock.cc:648] HybridClock initialized: now 1790385019927036 us; error 29 us; skew 500 ppm
Sep 26 01:10:19 dist-test-slave-7b22 krb5kdc[6446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385019, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:10:19.929569 6553 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260926 01:10:19.930059 6553 webserver.cc:460] Webserver started at http://127.0.98.1:43807/ using document root <none> and password file <none>
I20260926 01:10:19.930260 6553 fs_manager.cc:359] Metadata directory not provided
I20260926 01:10:19.930300 6553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:10:19.930399 6553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:10:19.931424 6553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/ts-0/data/instance:
uuid: "2264518cb0fe444da3e1606269628249"
format_stamp: "Formatted at 2026-09-26 01:10:19 on dist-test-slave-7b22"
server_key: "48d221961e9055483bd294f110dc6b34"
server_key_iv: "a6c14ae5682365af06cfad225051ae2c"
server_key_version: "encryptionkey@0"
I20260926 01:10:19.931732 6553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/ts-0/wal/instance:
uuid: "2264518cb0fe444da3e1606269628249"
format_stamp: "Formatted at 2026-09-26 01:10:19 on dist-test-slave-7b22"
server_key: "48d221961e9055483bd294f110dc6b34"
server_key_iv: "a6c14ae5682365af06cfad225051ae2c"
server_key_version: "encryptionkey@0"
I20260926 01:10:19.933064 6553 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 01:10:19.933889 6568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:19.934052 6553 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:10:19.934132 6553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/ts-0/wal
uuid: "2264518cb0fe444da3e1606269628249"
format_stamp: "Formatted at 2026-09-26 01:10:19 on dist-test-slave-7b22"
server_key: "48d221961e9055483bd294f110dc6b34"
server_key_iv: "a6c14ae5682365af06cfad225051ae2c"
server_key_version: "encryptionkey@0"
I20260926 01:10:19.934195 6553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:10:19.946738 6553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:10:19.948817 6553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:10:19.948932 6553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:10:19.949119 6553 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:10:19.949376 6553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:10:19.949404 6553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:19.949426 6553 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:10:19.949442 6553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:19.954993 6553 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35643
I20260926 01:10:19.955060 6681 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35643 every 8 connection(s)
I20260926 01:10:19.955348 6553 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/ts-0/data/info.pb
I20260926 01:10:19.958496 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 6553
I20260926 01:10:19.958611 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/ts-0/wal/instance
I20260926 01:10:19.958819 392 external_mini_cluster.cc:1428] Setting key 62f80bbc34ba7f6211f8bedb3af6411e
Sep 26 01:10:19 dist-test-slave-7b22 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385019, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 01:10:19.965237 6544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:19.956059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38919 (local address 127.0.98.62:45453)
0926 01:10:19.956107 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:19.956109 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:19.956329 (+ 220us) server_negotiation.cc:407] Connection header received
0926 01:10:19.956803 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:19.956804 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:19.956821 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:19.956872 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:19.957812 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:19.958404 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:19.959782 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:19.959995 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:19.960058 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:19.962373 (+ 2315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:10:19.962374 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:10:19.962374 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:10:19.962381 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:10:19.963672 (+ 1291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:19.964170 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:19.964171 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:19.964171 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:19.964231 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:19.964525 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:19.964526 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:19.964526 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:19.964677 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:10:19.964746 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:19.965080 (+ 334us) server_negotiation.cc:299] Negotiation successful
0926 01:10:19.965167 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 01:10:19.965478 6684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:19.956052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45453 (local address 127.0.98.1:38919)
0926 01:10:19.956210 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:19.956246 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:19.956709 (+ 463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:19.956941 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:19.956945 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:19.957505 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:10:19.957700 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:19.957706 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:19.958510 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:19.958511 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:19.959668 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:19.959672 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:19.960077 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:19.960078 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:19.960222 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:19.960750 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:10:19.960761 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:10:19.962274 (+ 1513us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:10:19.963762 (+ 1488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:10:19.963770 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:10:19.963773 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:10:19.964088 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:10:19.964293 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:10:19.964294 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:10:19.964294 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:10:19.964435 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:10:19.964747 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:10:19.964754 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:10:19.964940 (+ 186us) client_negotiation.cc:769] Sending connection context
0926 01:10:19.965061 (+ 121us) client_negotiation.cc:240] Negotiation successful
0926 01:10:19.965237 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":79,"threads_started":1}
I20260926 01:10:19.966107 6682 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45453
I20260926 01:10:19.966213 6682 heartbeater.cc:464] Registering TS with master...
I20260926 01:10:19.966418 6682 heartbeater.cc:511] Master 127.0.98.62:45453 requested a full tablet report, sending...
I20260926 01:10:19.966964 6491 ts_manager.cc:194] Registered new tserver with Master: 2264518cb0fe444da3e1606269628249 (127.0.98.1:35643)
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260926 01:10:19.967720 6491 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:38919
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260926 01:10:19.985646 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45453
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:10:20.065478 6687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:10:20.065691 6687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:10:20.065713 6687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:10:20.066900 6687 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:10:20.066951 6687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:10:20.066978 6687 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 01:10:20.068418 6687 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45453
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6687
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:10:20.068609 6687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:10:20.068791 6687 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:10:20.073836 6693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:10:20.073813 6695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:10:20.073926 6692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:10:20.074116 6687 server_base.cc:1034] running on GCE node
I20260926 01:10:20.074350 6687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:10:20.074661 6687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:10:20.075832 6687 hybrid_clock.cc:648] HybridClock initialized: now 1790385020075796 us; error 49 us; skew 500 ppm
Sep 26 01:10:20 dist-test-slave-7b22 krb5kdc[6446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385020, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:10:20.078434 6687 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260926 01:10:20.078907 6687 webserver.cc:460] Webserver started at http://127.0.98.2:38159/ using document root <none> and password file <none>
I20260926 01:10:20.079123 6687 fs_manager.cc:359] Metadata directory not provided
I20260926 01:10:20.079172 6687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:10:20.079313 6687 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:10:20.080467 6687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/ts-1/data/instance:
uuid: "1ffe4fc932934431b944d78dfe891007"
format_stamp: "Formatted at 2026-09-26 01:10:20 on dist-test-slave-7b22"
server_key: "5d16c6db766df4828323c7d06db0c4e8"
server_key_iv: "eb7998c144aed7bd52a253cf01920a32"
server_key_version: "encryptionkey@0"
I20260926 01:10:20.080827 6687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/ts-1/wal/instance:
uuid: "1ffe4fc932934431b944d78dfe891007"
format_stamp: "Formatted at 2026-09-26 01:10:20 on dist-test-slave-7b22"
server_key: "5d16c6db766df4828323c7d06db0c4e8"
server_key_iv: "eb7998c144aed7bd52a253cf01920a32"
server_key_version: "encryptionkey@0"
I20260926 01:10:20.082212 6687 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 01:10:20.083253 6702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:20.083460 6687 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 01:10:20.083556 6687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/ts-1/wal
uuid: "1ffe4fc932934431b944d78dfe891007"
format_stamp: "Formatted at 2026-09-26 01:10:20 on dist-test-slave-7b22"
server_key: "5d16c6db766df4828323c7d06db0c4e8"
server_key_iv: "eb7998c144aed7bd52a253cf01920a32"
server_key_version: "encryptionkey@0"
I20260926 01:10:20.083621 6687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:10:20.109282 6687 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:10:20.111691 6687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:10:20.111848 6687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:10:20.112089 6687 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:10:20.112401 6687 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:10:20.112437 6687 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:20.112466 6687 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:10:20.112488 6687 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:20.118284 6687 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39297
I20260926 01:10:20.118351 6815 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39297 every 8 connection(s)
I20260926 01:10:20.118652 6687 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/ts-1/data/info.pb
I20260926 01:10:20.120707 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 6687
I20260926 01:10:20.120814 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/ts-1/wal/instance
I20260926 01:10:20.121014 392 external_mini_cluster.cc:1428] Setting key 773cecf15c47dea8a909edfa479aeec2
Sep 26 01:10:20 dist-test-slave-7b22 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385020, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260926 01:10:20.128535 6544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:20.119382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34649 (local address 127.0.98.62:45453)
0926 01:10:20.119438 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:20.119441 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:20.119614 (+ 173us) server_negotiation.cc:407] Connection header received
0926 01:10:20.120089 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:20.120090 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:20.120100 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:20.120145 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:20.121314 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:20.121866 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:20.122899 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:20.123112 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:20.123182 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:20.125581 (+ 2399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:10:20.125583 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:10:20.125583 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:10:20.125592 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:10:20.127156 (+ 1564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:20.127586 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:20.127587 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:20.127588 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:20.127642 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:20.127862 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:20.127863 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:20.127863 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:20.127982 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:10:20.128039 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:20.128319 (+ 280us) server_negotiation.cc:299] Negotiation successful
0926 01:10:20.128429 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 01:10:20.128714 6818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:20.119351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45453 (local address 127.0.98.2:34649)
0926 01:10:20.119509 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:20.119543 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:20.120009 (+ 466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:20.120207 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:20.120210 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:20.120933 (+ 723us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:10:20.121217 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:20.121224 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:20.121952 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:20.121953 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:20.122799 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:20.122802 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:20.123201 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:20.123203 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:20.123334 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:20.123878 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:10:20.123887 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:10:20.125482 (+ 1595us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:10:20.127274 (+ 1792us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:10:20.127283 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:10:20.127286 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:10:20.127514 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:10:20.127696 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:10:20.127696 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:10:20.127697 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:10:20.127806 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:10:20.128035 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:10:20.128042 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:10:20.128187 (+ 145us) client_negotiation.cc:769] Sending connection context
0926 01:10:20.128299 (+ 112us) client_negotiation.cc:240] Negotiation successful
0926 01:10:20.128488 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":91,"threads_started":1}
I20260926 01:10:20.129238 6816 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45453
I20260926 01:10:20.129338 6816 heartbeater.cc:464] Registering TS with master...
I20260926 01:10:20.129547 6816 heartbeater.cc:511] Master 127.0.98.62:45453 requested a full tablet report, sending...
I20260926 01:10:20.130035 6491 ts_manager.cc:194] Registered new tserver with Master: 1ffe4fc932934431b944d78dfe891007 (127.0.98.2:39297)
I20260926 01:10:20.130467 6491 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:34649
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260926 01:10:20.145678 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45453
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:10:20.225505 6821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:10:20.225798 6821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:10:20.225858 6821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:10:20.227555 6821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:10:20.227614 6821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:10:20.227654 6821 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 01:10:20.229329 6821 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45453
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6821
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:10:20.229542 6821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:10:20.229734 6821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 01:10:20.234638 6821 server_base.cc:1034] running on GCE node
W20260926 01:10:20.234593 6829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:10:20.234588 6827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:10:20.234579 6826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:10:20.235060 6821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:10:20.235294 6821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:10:20.236423 6821 hybrid_clock.cc:648] HybridClock initialized: now 1790385020236415 us; error 30 us; skew 500 ppm
Sep 26 01:10:20 dist-test-slave-7b22 krb5kdc[6446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385020, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:10:20.238901 6821 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260926 01:10:20.239341 6821 webserver.cc:460] Webserver started at http://127.0.98.3:36287/ using document root <none> and password file <none>
I20260926 01:10:20.239539 6821 fs_manager.cc:359] Metadata directory not provided
I20260926 01:10:20.239588 6821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:10:20.239712 6821 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:10:20.240787 6821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/ts-2/data/instance:
uuid: "72a12d78947e445c85a8d6222cc136af"
format_stamp: "Formatted at 2026-09-26 01:10:20 on dist-test-slave-7b22"
server_key: "fb7f7d6f7d8015dea249e16f1437c8a1"
server_key_iv: "43588d170860f4bdd29d12ecea839a22"
server_key_version: "encryptionkey@0"
I20260926 01:10:20.241127 6821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/ts-2/wal/instance:
uuid: "72a12d78947e445c85a8d6222cc136af"
format_stamp: "Formatted at 2026-09-26 01:10:20 on dist-test-slave-7b22"
server_key: "fb7f7d6f7d8015dea249e16f1437c8a1"
server_key_iv: "43588d170860f4bdd29d12ecea839a22"
server_key_version: "encryptionkey@0"
I20260926 01:10:20.242487 6821 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 01:10:20.243235 6836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:20.243394 6821 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 01:10:20.243479 6821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/ts-2/wal
uuid: "72a12d78947e445c85a8d6222cc136af"
format_stamp: "Formatted at 2026-09-26 01:10:20 on dist-test-slave-7b22"
server_key: "fb7f7d6f7d8015dea249e16f1437c8a1"
server_key_iv: "43588d170860f4bdd29d12ecea839a22"
server_key_version: "encryptionkey@0"
I20260926 01:10:20.243541 6821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:10:20.261315 6821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:10:20.263477 6821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:10:20.263592 6821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:10:20.263759 6821 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:10:20.263998 6821 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:10:20.264026 6821 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:20.264046 6821 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:10:20.264060 6821 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:20.269371 6821 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43231
I20260926 01:10:20.269423 6949 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43231 every 8 connection(s)
I20260926 01:10:20.269726 6821 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/ts-2/data/info.pb
I20260926 01:10:20.270681 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 6821
I20260926 01:10:20.270781 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/ts-2/wal/instance
I20260926 01:10:20.270946 392 external_mini_cluster.cc:1428] Setting key d155574557aa3ff48863cb453e1de28b
Sep 26 01:10:20 dist-test-slave-7b22 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385020, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 01:10:20.278853 6544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:20.270420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40439 (local address 127.0.98.62:45453)
0926 01:10:20.270475 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:20.270497 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:20.270650 (+ 153us) server_negotiation.cc:407] Connection header received
0926 01:10:20.271277 (+ 627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:20.271278 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:20.271291 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:20.271344 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:20.272340 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:20.272824 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:20.274164 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:20.274351 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:20.274411 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:20.276557 (+ 2146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:10:20.276558 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:10:20.276559 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:10:20.276566 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:10:20.277749 (+ 1183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:20.278142 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:20.278142 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:20.278143 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:20.278189 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:20.278378 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:20.278379 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:20.278379 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:20.278475 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:10:20.278520 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:20.278729 (+ 209us) server_negotiation.cc:299] Negotiation successful
0926 01:10:20.278787 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 01:10:20.279055 6952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:20.270393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45453 (local address 127.0.98.3:40439)
0926 01:10:20.270530 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:20.270575 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:20.271173 (+ 598us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:20.271414 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:20.271417 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:20.272052 (+ 635us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:10:20.272249 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:20.272253 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:20.272922 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:20.272923 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:20.274084 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:20.274087 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:20.274421 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:20.274422 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:20.274521 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:20.275079 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:10:20.275089 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:10:20.276473 (+ 1384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:10:20.277853 (+ 1380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:10:20.277859 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:10:20.277862 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:10:20.278083 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:10:20.278236 (+ 153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:10:20.278237 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:10:20.278237 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:10:20.278332 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:10:20.278515 (+ 183us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:10:20.278518 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:10:20.278633 (+ 115us) client_negotiation.cc:769] Sending connection context
0926 01:10:20.278710 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 01:10:20.278841 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":64,"threads_started":1}
I20260926 01:10:20.279484 6950 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45453
I20260926 01:10:20.279573 6950 heartbeater.cc:464] Registering TS with master...
I20260926 01:10:20.279749 6950 heartbeater.cc:511] Master 127.0.98.62:45453 requested a full tablet report, sending...
I20260926 01:10:20.280108 6491 ts_manager.cc:194] Registered new tserver with Master: 72a12d78947e445c85a8d6222cc136af (127.0.98.3:43231)
I20260926 01:10:20.280472 6491 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:40439
I20260926 01:10:20.282660 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/krb5kdc/oryx_127.0.98.61.keytab.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/krb5kdc/oryx_127.0.98.61.keytab.
I20260926 01:10:20.308993 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:42317
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45453,127.0.98.61:42317
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:10:20.389400 6955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:10:20.389644 6955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:10:20.389676 6955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:10:20.390990 6955 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:10:20.391040 6955 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:10:20.391059 6955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:10:20.391076 6955 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:10:20.391093 6955 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:10:20.393003 6955 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45453,127.0.98.61:42317
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:42317
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6955
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
W20260926 01:10:20.393146 6955 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.98.62:45453,127.0.98.61:42317'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260926 01:10:20.393317 6955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:10:20.393538 6955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:10:20.398275 6960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:10:20.398279 6963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:10:20.398293 6961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:10:20.398571 6955 server_base.cc:1034] running on GCE node
I20260926 01:10:20.398799 6955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:10:20.399077 6955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:10:20.400239 6955 hybrid_clock.cc:648] HybridClock initialized: now 1790385020400222 us; error 31 us; skew 500 ppm
Sep 26 01:10:20 dist-test-slave-7b22 krb5kdc[6446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385020, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:10:20.402799 6955 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260926 01:10:20.403280 6955 webserver.cc:460] Webserver started at http://127.0.98.61:43113/ using document root <none> and password file <none>
I20260926 01:10:20.403477 6955 fs_manager.cc:359] Metadata directory not provided
I20260926 01:10:20.403520 6955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:10:20.403647 6955 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:10:20.404708 6955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/master-1/data/instance:
uuid: "8971f3cf560c49ae9dd58727c3b669c4"
format_stamp: "Formatted at 2026-09-26 01:10:20 on dist-test-slave-7b22"
server_key: "57b663ef5023224b8aaa9b296ee8774d"
server_key_iv: "53c64e1d4ed05724b95a5203f58342d7"
server_key_version: "encryptionkey@0"
I20260926 01:10:20.405026 6955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/master-1/wal/instance:
uuid: "8971f3cf560c49ae9dd58727c3b669c4"
format_stamp: "Formatted at 2026-09-26 01:10:20 on dist-test-slave-7b22"
server_key: "57b663ef5023224b8aaa9b296ee8774d"
server_key_iv: "53c64e1d4ed05724b95a5203f58342d7"
server_key_version: "encryptionkey@0"
I20260926 01:10:20.406368 6955 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 01:10:20.407094 6970 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:20.407258 6955 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 01:10:20.407342 6955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/master-1/wal
uuid: "8971f3cf560c49ae9dd58727c3b669c4"
format_stamp: "Formatted at 2026-09-26 01:10:20 on dist-test-slave-7b22"
server_key: "57b663ef5023224b8aaa9b296ee8774d"
server_key_iv: "53c64e1d4ed05724b95a5203f58342d7"
server_key_version: "encryptionkey@0"
I20260926 01:10:20.407428 6955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:10:20.425731 6955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:10:20.427807 6955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:10:20.427954 6955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:10:20.431818 6955 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:42317
I20260926 01:10:20.431851 7022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:42317 every 8 connection(s)
I20260926 01:10:20.432193 6955 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/master-1/data/info.pb
I20260926 01:10:20.433223 7023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:10:20.433593 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 6955
I20260926 01:10:20.435031 7023 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45453 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 01:10:20 dist-test-slave-7b22 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385020, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 26 01:10:20 dist-test-slave-7b22 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385019, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260926 01:10:20.443799 6544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:20.435768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44370 (local address 127.0.98.62:45453)
0926 01:10:20.435834 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:20.435836 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:20.435974 (+ 138us) server_negotiation.cc:407] Connection header received
0926 01:10:20.436036 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:20.436036 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:20.436045 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:20.436091 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:20.436646 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:20.437187 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:20.438216 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:20.438439 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:20.438504 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:20.440523 (+ 2019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:10:20.440525 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:10:20.440525 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:10:20.440532 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:10:20.442195 (+ 1663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:20.442646 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:20.442648 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:20.442648 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:20.442725 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:20.442965 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:20.442966 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:20.442966 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:20.443197 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:10:20.443274 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:20.443609 (+ 335us) server_negotiation.cc:299] Negotiation successful
0926 01:10:20.443698 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 01:10:20.443997 7027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:20.435695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45453 (local address 127.0.0.1:44370)
0926 01:10:20.435876 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:20.435916 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:20.435981 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:20.436106 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:20.436108 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:20.436437 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:10:20.436570 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:20.436576 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:20.437285 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:20.437287 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:20.438142 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:20.438144 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:20.438513 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:20.438513 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:20.438566 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:20.439082 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:10:20.439090 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:10:20.440439 (+ 1349us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:10:20.442278 (+ 1839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:10:20.442283 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:10:20.442286 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:10:20.442575 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:10:20.442796 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:10:20.442797 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:10:20.442797 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:10:20.442902 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:10:20.443264 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:10:20.443269 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:10:20.443463 (+ 194us) client_negotiation.cc:769] Sending connection context
0926 01:10:20.443552 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 01:10:20.443709 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"mutex_wait_us":1,"thread_start_us":127,"threads_started":1}
I20260926 01:10:20.443996 7026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:20.434300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37826 (local address 127.0.98.61:42317)
0926 01:10:20.434455 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:20.434462 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:20.434487 (+ 25us) server_negotiation.cc:407] Connection header received
0926 01:10:20.434897 (+ 410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:20.434904 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:20.434951 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:20.435066 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:20.435781 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:20.436970 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:20.437840 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:20.438330 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:20.438406 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:20.440982 (+ 2576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:10:20.440989 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:10:20.440990 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:10:20.440996 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:10:20.442545 (+ 1549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:20.442999 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:20.443002 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:20.443003 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:20.443045 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:20.443253 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:20.443254 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:20.443254 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:20.443465 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:10:20.443597 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:20.443805 (+ 208us) server_negotiation.cc:299] Negotiation successful
0926 01:10:20.443888 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":105,"thread_start_us":57,"threads_started":1}
I20260926 01:10:20.444564 7023 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42317 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 01:10:20 dist-test-slave-7b22 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385020, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260926 01:10:20.450760 7026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:20.444880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37840 (local address 127.0.98.61:42317)
0926 01:10:20.444913 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:20.444915 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:20.445022 (+ 107us) server_negotiation.cc:407] Connection header received
0926 01:10:20.445085 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:20.445085 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:20.445093 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:20.445126 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:20.445358 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:20.445841 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:20.446566 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:20.446749 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:20.446792 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:20.448474 (+ 1682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:10:20.448476 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:10:20.448476 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:10:20.448482 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:10:20.449687 (+ 1205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:20.450052 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:20.450053 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:20.450054 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:20.450102 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:20.450232 (+ 130us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:20.450232 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:20.450233 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:20.450330 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:10:20.450374 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:20.450551 (+ 177us) server_negotiation.cc:299] Negotiation successful
0926 01:10:20.450676 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":59,"server-negotiator.queue_time_us":15}
I20260926 01:10:20.450810 7027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:20.444826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42317 (local address 127.0.0.1:37840)
0926 01:10:20.444984 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:20.444995 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:20.445036 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:20.445137 (+ 101us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:20.445138 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:20.445264 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:10:20.445311 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:20.445312 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:20.445920 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:20.445920 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:20.446499 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:20.446501 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:20.446797 (+ 296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:20.446798 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:20.446844 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:20.447210 (+ 366us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:10:20.447216 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:10:20.448377 (+ 1161us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:10:20.449786 (+ 1409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:10:20.449787 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:10:20.449787 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:10:20.449995 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:10:20.450150 (+ 155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:10:20.450151 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:10:20.450151 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:10:20.450199 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:10:20.450397 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:10:20.450398 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:10:20.450438 (+ 40us) client_negotiation.cc:769] Sending connection context
0926 01:10:20.450513 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 01:10:20.450615 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144}
I20260926 01:10:20.452354 7023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8971f3cf560c49ae9dd58727c3b669c4: Bootstrap starting.
I20260926 01:10:20.453112 7023 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8971f3cf560c49ae9dd58727c3b669c4: Neither blocks nor log segments found. Creating new log.
I20260926 01:10:20.453336 7023 log.cc:824] T 00000000000000000000000000000000 P 8971f3cf560c49ae9dd58727c3b669c4: Log is configured to *not* fsync() on all Append() calls
I20260926 01:10:20.454085 7023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8971f3cf560c49ae9dd58727c3b669c4: No bootstrap required, opened a new log
I20260926 01:10:20.455435 7023 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8971f3cf560c49ae9dd58727c3b669c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5afbc257ec7449b189377b1ae0cbebd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45453 } } peers { permanent_uuid: "8971f3cf560c49ae9dd58727c3b669c4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42317 } }
I20260926 01:10:20.455566 7023 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8971f3cf560c49ae9dd58727c3b669c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:10:20.455587 7023 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8971f3cf560c49ae9dd58727c3b669c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8971f3cf560c49ae9dd58727c3b669c4, State: Initialized, Role: FOLLOWER
I20260926 01:10:20.455742 7023 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8971f3cf560c49ae9dd58727c3b669c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5afbc257ec7449b189377b1ae0cbebd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45453 } } peers { permanent_uuid: "8971f3cf560c49ae9dd58727c3b669c4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42317 } }
I20260926 01:10:20.456022 7023 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8971f3cf560c49ae9dd58727c3b669c4 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:10:20.456008 7029 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8971f3cf560c49ae9dd58727c3b669c4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5afbc257ec7449b189377b1ae0cbebd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45453 } } peers { permanent_uuid: "8971f3cf560c49ae9dd58727c3b669c4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42317 } } }
I20260926 01:10:20.456110 7029 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8971f3cf560c49ae9dd58727c3b669c4 [sys.catalog]: This master's current role is: FOLLOWER
W20260926 01:10:20.457512 7040 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8971f3cf560c49ae9dd58727c3b669c4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 01:10:20.457574 7040 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 01:10:20.459242 6955 master_runner.cc:428] Detected that this master 8971f3cf560c49ae9dd58727c3b669c4 is joining an existing cluster
I20260926 01:10:20.459288 6955 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.61:42317
I20260926 01:10:20.467422 7026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:20.460513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37850 (local address 127.0.98.61:42317)
0926 01:10:20.460541 (+ 28us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:20.460542 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:20.460632 (+ 90us) server_negotiation.cc:407] Connection header received
0926 01:10:20.460708 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:20.460709 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:20.460717 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:20.460755 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:20.461155 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:20.461872 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:20.462948 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:20.463126 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:20.463191 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:20.464174 (+ 983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:10:20.464175 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:10:20.464175 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:10:20.464183 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:10:20.465761 (+ 1578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:20.466138 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:20.466139 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:20.466140 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:20.466189 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:20.466864 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:20.466865 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:20.466865 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:20.467013 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:10:20.467072 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:20.467222 (+ 150us) server_negotiation.cc:299] Negotiation successful
0926 01:10:20.467333 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":51,"server-negotiator.queue_time_us":14}
I20260926 01:10:20.467429 7047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:20.460472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42317 (local address 127.0.0.1:37850)
0926 01:10:20.460577 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:20.460586 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:20.460652 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:20.460770 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:20.460771 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:20.461018 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:10:20.461079 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:20.461081 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:20.461946 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:20.461948 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:20.462852 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:20.462855 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:20.463233 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:20.463234 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:20.463293 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:20.463712 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:10:20.463717 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:10:20.464094 (+ 377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:10:20.465858 (+ 1764us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:10:20.465859 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:10:20.465860 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:10:20.466055 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:10:20.466240 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:10:20.466241 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:10:20.466241 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:10:20.466296 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:10:20.467095 (+ 799us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:10:20.467095 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:10:20.467130 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 01:10:20.467188 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 01:10:20.467286 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":56,"threads_started":1}
I20260926 01:10:20.467967 6544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:20.460286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44382 (local address 127.0.98.62:45453)
0926 01:10:20.460326 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:20.460328 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:20.460539 (+ 211us) server_negotiation.cc:407] Connection header received
0926 01:10:20.460564 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:20.460565 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:20.460575 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:20.460628 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:20.461056 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:20.461866 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:20.463945 (+ 2079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:20.464235 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:20.464293 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:20.465284 (+ 991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:10:20.465285 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:10:20.465286 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:10:20.465293 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:10:20.466786 (+ 1493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:20.467119 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:20.467120 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:20.467121 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:20.467189 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:20.467405 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:20.467406 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:20.467406 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:20.467535 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:10:20.467574 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:20.467792 (+ 218us) server_negotiation.cc:299] Negotiation successful
0926 01:10:20.467874 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 01:10:20.468251 7046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:20.460249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45453 (local address 127.0.0.1:44382)
0926 01:10:20.460401 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:20.460410 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:20.460465 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:20.460633 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:20.460634 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:20.460915 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:10:20.460986 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:20.460989 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:20.462959 (+ 1970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:20.462959 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:20.463854 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:20.463856 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:20.464294 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:20.464295 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:20.464354 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:20.464817 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:10:20.464822 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:10:20.465210 (+ 388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:10:20.466851 (+ 1641us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:10:20.466852 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:10:20.466852 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:10:20.467044 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:10:20.467273 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:10:20.467274 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:10:20.467274 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:10:20.467337 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:10:20.467617 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:10:20.467618 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:10:20.467667 (+ 49us) client_negotiation.cc:769] Sending connection context
0926 01:10:20.468053 (+ 386us) client_negotiation.cc:240] Negotiation successful
0926 01:10:20.468148 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":93,"threads_started":1}
Sep 26 01:10:20 dist-test-slave-7b22 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385019, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260926 01:10:20.477656 7026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:20.470411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37854 (local address 127.0.98.61:42317)
0926 01:10:20.470446 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:20.470448 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:20.470589 (+ 141us) server_negotiation.cc:407] Connection header received
0926 01:10:20.470675 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:20.470676 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:20.470685 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:20.470730 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:20.471178 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:20.471653 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:20.472415 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:20.472612 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:20.472664 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:20.475112 (+ 2448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:10:20.475113 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:10:20.475114 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:10:20.475121 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:10:20.476513 (+ 1392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:20.476909 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:20.476910 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:20.476910 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:20.476967 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:20.477129 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:20.477130 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:20.477130 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:20.477243 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:10:20.477309 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:20.477465 (+ 156us) server_negotiation.cc:299] Negotiation successful
0926 01:10:20.477559 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260926 01:10:20.477674 7048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:20.470368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42317 (local address 127.0.0.1:37854)
0926 01:10:20.470500 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:20.470532 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:20.470602 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:20.470791 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:20.470796 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:20.471055 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:10:20.471107 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:20.471109 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:20.471739 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:20.471740 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:20.472339 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:20.472342 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:20.472671 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:20.472672 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:20.472753 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:20.473275 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:10:20.473280 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:10:20.474990 (+ 1710us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:10:20.476613 (+ 1623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:10:20.476623 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:10:20.476626 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:10:20.476843 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:10:20.477023 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:10:20.477024 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:10:20.477025 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:10:20.477081 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:10:20.477295 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:10:20.477303 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:10:20.477362 (+ 59us) client_negotiation.cc:769] Sending connection context
0926 01:10:20.477435 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 01:10:20.477564 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":81,"threads_started":1}
I20260926 01:10:20.478302 6491 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.61:42317: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "8971f3cf560c49ae9dd58727c3b669c4"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.61"
port: 42317
}
attrs {
promote: true
}
}
dest_uuid: "5afbc257ec7449b189377b1ae0cbebd5"
cas_config_opid_index: -1
I20260926 01:10:20.478646 6491 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5afbc257ec7449b189377b1ae0cbebd5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5afbc257ec7449b189377b1ae0cbebd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45453 } } peers { permanent_uuid: "8971f3cf560c49ae9dd58727c3b669c4" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 42317 } attrs { promote: true } }
I20260926 01:10:20.479115 7050 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5afbc257ec7449b189377b1ae0cbebd5 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 8971f3cf560c49ae9dd58727c3b669c4 (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5afbc257ec7449b189377b1ae0cbebd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45453 } } peers { permanent_uuid: "8971f3cf560c49ae9dd58727c3b669c4" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 42317 } attrs { promote: true } } }
I20260926 01:10:20.479519 6997 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8971f3cf560c49ae9dd58727c3b669c4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:10:20.480294 7050 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.61:42317
I20260926 01:10:20.480397 6997 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8971f3cf560c49ae9dd58727c3b669c4 [term 1 FOLLOWER]: Refusing update from remote peer 5afbc257ec7449b189377b1ae0cbebd5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260926 01:10:20.480486 7049 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5afbc257ec7449b189377b1ae0cbebd5 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "5afbc257ec7449b189377b1ae0cbebd5" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5afbc257ec7449b189377b1ae0cbebd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45453 } } peers { permanent_uuid: "8971f3cf560c49ae9dd58727c3b669c4" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 42317 } attrs { promote: true } } }
I20260926 01:10:20.480566 7049 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5afbc257ec7449b189377b1ae0cbebd5 [sys.catalog]: This master's current role is: LEADER
I20260926 01:10:20.480613 6955 master.cc:537] Master@127.0.98.61:42317 shutting down...
I20260926 01:10:20.480702 7050 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5afbc257ec7449b189377b1ae0cbebd5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8971f3cf560c49ae9dd58727c3b669c4" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 42317 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 01:10:20.481370 6477 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260926 01:10:20.481858 6477 consensus_peers.cc:487] T 00000000000000000000000000000000 P 5afbc257ec7449b189377b1ae0cbebd5 -> Peer 8971f3cf560c49ae9dd58727c3b669c4 (127.0.98.61:42317): Couldn't send request to peer 8971f3cf560c49ae9dd58727c3b669c4. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260926 01:10:20.483243 6955 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 8971f3cf560c49ae9dd58727c3b669c4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:10:20.483335 6955 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 8971f3cf560c49ae9dd58727c3b669c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:10:20.483387 6955 tablet_replica.cc:330] T 00000000000000000000000000000000 P 8971f3cf560c49ae9dd58727c3b669c4: stopping tablet replica
I20260926 01:10:20.496598 6955 master.cc:552] Master@127.0.98.61:42317 shutdown complete.
I20260926 01:10:20.496661 6955 master_runner.cc:305] Clearing existing system tablet
I20260926 01:10:20.496802 6955 fs_manager.cc:359] Metadata directory not provided
I20260926 01:10:20.496842 6955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:10:20.497583 6955 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 01:10:20.498979 7054 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:20.499127 6955 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:10:20.499208 6955 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 01:10:20.499236 6955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/master-1/wal
uuid: "8971f3cf560c49ae9dd58727c3b669c4"
format_stamp: "Formatted at 2026-09-26 01:10:20 on dist-test-slave-7b22"
server_key: "57b663ef5023224b8aaa9b296ee8774d"
server_key_iv: "53c64e1d4ed05724b95a5203f58342d7"
server_key_version: "encryptionkey@0"
I20260926 01:10:20.500298 6955 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 8971f3cf560c49ae9dd58727c3b669c4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:10:20.501960 6955 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 8971f3cf560c49ae9dd58727c3b669c4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260926 01:10:20.502043 6955 log.cc:1196] T 00000000000000000000000000000000 P 8971f3cf560c49ae9dd58727c3b669c4: Deleting WAL directory at /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260926 01:10:20.502242 6955 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 8971f3cf560c49ae9dd58727c3b669c4: Deleting consensus metadata
I20260926 01:10:20.502359 6955 master_runner.cc:315] Copying system tablet from 127.0.98.62:45453
I20260926 01:10:20.503014 6955 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 8971f3cf560c49ae9dd58727c3b669c4: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:45453
I20260926 01:10:20.509503 6544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:20.503533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44392 (local address 127.0.98.62:45453)
0926 01:10:20.503579 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:20.503581 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:20.503590 (+ 9us) server_negotiation.cc:407] Connection header received
0926 01:10:20.503781 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:20.503782 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:20.503796 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:20.503865 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:20.504266 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:20.505034 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:20.505808 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:20.506056 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:20.506117 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:20.507122 (+ 1005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:10:20.507123 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:10:20.507124 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:10:20.507132 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:10:20.508425 (+ 1293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:20.508786 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:20.508787 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:20.508787 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:20.508841 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:20.509002 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:20.509003 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:20.509003 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:20.509119 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:10:20.509176 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:20.509326 (+ 150us) server_negotiation.cc:299] Negotiation successful
0926 01:10:20.509404 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 01:10:20.509523 7060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:20.503349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45453 (local address 127.0.0.1:44392)
0926 01:10:20.503519 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:20.503530 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:20.503585 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:20.503892 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:20.503893 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:20.504110 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:10:20.504178 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:20.504180 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:20.505120 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:20.505121 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:20.505716 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:20.505718 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:20.506123 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:20.506124 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:20.506195 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:20.506702 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:10:20.506707 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:10:20.507018 (+ 311us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:10:20.508518 (+ 1500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:10:20.508520 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:10:20.508520 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:10:20.508696 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:10:20.508895 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:10:20.508896 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:10:20.508896 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:10:20.508953 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:10:20.509170 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:10:20.509170 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:10:20.509232 (+ 62us) client_negotiation.cc:769] Sending connection context
0926 01:10:20.509291 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 01:10:20.509406 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":78,"threads_started":1}
I20260926 01:10:20.509953 6511 tablet_copy_service.cc:140] P 5afbc257ec7449b189377b1ae0cbebd5: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 8971f3cf560c49ae9dd58727c3b669c4 ({username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:44392)
I20260926 01:10:20.510076 6511 tablet_copy_service.cc:161] P 5afbc257ec7449b189377b1ae0cbebd5: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 8971f3cf560c49ae9dd58727c3b669c4 at {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:44392: session id = 8971f3cf560c49ae9dd58727c3b669c4-00000000000000000000000000000000
I20260926 01:10:20.510921 6511 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 5afbc257ec7449b189377b1ae0cbebd5: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 01:10:20.511641 6955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260926 01:10:20.512727 6955 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 8971f3cf560c49ae9dd58727c3b669c4: tablet copy: Starting download of 0 data blocks...
I20260926 01:10:20.512872 6955 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 8971f3cf560c49ae9dd58727c3b669c4: tablet copy: Starting download of 1 WAL segments...
I20260926 01:10:20.514042 6955 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 8971f3cf560c49ae9dd58727c3b669c4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 01:10:20.514943 6511 tablet_copy_service.cc:342] P 5afbc257ec7449b189377b1ae0cbebd5: Request end of tablet copy session 8971f3cf560c49ae9dd58727c3b669c4-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:44392
I20260926 01:10:20.515012 6511 tablet_copy_service.cc:433] P 5afbc257ec7449b189377b1ae0cbebd5: ending tablet copy session 8971f3cf560c49ae9dd58727c3b669c4-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 8971f3cf560c49ae9dd58727c3b669c4
I20260926 01:10:20.517217 6955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:10:20.518369 7062 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:10:20.518384 7061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:10:20.518411 7064 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:10:20.522010 6955 server_base.cc:1034] running on GCE node
I20260926 01:10:20.522146 6955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:10:20.522302 6955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:10:20.523443 6955 hybrid_clock.cc:648] HybridClock initialized: now 1790385020523437 us; error 38 us; skew 500 ppm
Sep 26 01:10:20 dist-test-slave-7b22 krb5kdc[6446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385020, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:10:20.524863 6955 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260926 01:10:20.525180 6955 webserver.cc:460] Webserver started at http://127.0.98.61:35731/ using document root <none> and password file <none>
I20260926 01:10:20.525300 6955 fs_manager.cc:359] Metadata directory not provided
I20260926 01:10:20.525336 6955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:10:20.526088 6955 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:10:20.526676 7071 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:20.526880 6955 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 01:10:20.526927 6955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/master-1/wal
uuid: "8971f3cf560c49ae9dd58727c3b669c4"
format_stamp: "Formatted at 2026-09-26 01:10:20 on dist-test-slave-7b22"
server_key: "57b663ef5023224b8aaa9b296ee8774d"
server_key_iv: "53c64e1d4ed05724b95a5203f58342d7"
server_key_version: "encryptionkey@0"
I20260926 01:10:20.526989 6955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:10:20.541970 6955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:10:20.543929 6955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:10:20.547370 6955 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:42317
I20260926 01:10:20.547386 7123 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:42317 every 8 connection(s)
I20260926 01:10:20.547653 6955 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/master-1/data/info.pb
I20260926 01:10:20.548296 7124 sys_catalog.cc:263] Verifying existing consensus state
I20260926 01:10:20.548712 7124 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8971f3cf560c49ae9dd58727c3b669c4: Bootstrap starting.
I20260926 01:10:20.551047 7124 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8971f3cf560c49ae9dd58727c3b669c4: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 01:10:20.551229 7124 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8971f3cf560c49ae9dd58727c3b669c4: Bootstrap complete.
I20260926 01:10:20.551424 7124 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8971f3cf560c49ae9dd58727c3b669c4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5afbc257ec7449b189377b1ae0cbebd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45453 } } peers { permanent_uuid: "8971f3cf560c49ae9dd58727c3b669c4" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 42317 } attrs { promote: true } }
I20260926 01:10:20.551493 7124 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8971f3cf560c49ae9dd58727c3b669c4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8971f3cf560c49ae9dd58727c3b669c4, State: Initialized, Role: LEARNER
I20260926 01:10:20.551517 7124 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8971f3cf560c49ae9dd58727c3b669c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5afbc257ec7449b189377b1ae0cbebd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45453 } } peers { permanent_uuid: "8971f3cf560c49ae9dd58727c3b669c4" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 42317 } attrs { promote: true } }
I20260926 01:10:20.551673 7124 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8971f3cf560c49ae9dd58727c3b669c4 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:10:20.551702 7127 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8971f3cf560c49ae9dd58727c3b669c4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5afbc257ec7449b189377b1ae0cbebd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45453 } } peers { permanent_uuid: "8971f3cf560c49ae9dd58727c3b669c4" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 42317 } attrs { promote: true } } }
I20260926 01:10:20.551831 7127 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8971f3cf560c49ae9dd58727c3b669c4 [sys.catalog]: This master's current role is: LEARNER
I20260926 01:10:20.553130 7138 catalog_manager.cc:1259] Loaded cluster ID: d16d612222b24a05bd8871dd9f254e18
I20260926 01:10:20.553241 7138 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8971f3cf560c49ae9dd58727c3b669c4: loading cluster ID for follower catalog manager: success
I20260926 01:10:20.554147 7138 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8971f3cf560c49ae9dd58727c3b669c4: acquiring CA information for follower catalog manager: success
I20260926 01:10:20.554446 7138 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8971f3cf560c49ae9dd58727c3b669c4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 01:10:20.602640 7139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:20.596535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37892 (local address 127.0.98.61:42317)
0926 01:10:20.596671 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:20.596673 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:20.596679 (+ 6us) server_negotiation.cc:407] Connection header received
0926 01:10:20.596703 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:20.596704 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:20.596716 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:20.596779 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:20.597319 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:20.597988 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:20.598690 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:20.598878 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:20.598934 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:20.600123 (+ 1189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:10:20.600125 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:10:20.600125 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:10:20.600132 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:10:20.601520 (+ 1388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:20.601926 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:20.601928 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:20.601928 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:20.601979 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:20.602150 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:20.602151 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:20.602151 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:20.602268 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:10:20.602353 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:20.602461 (+ 108us) server_negotiation.cc:299] Negotiation successful
0926 01:10:20.602542 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":74,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.60@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/krb5kdc/oryx_127.0.98.60.keytab.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/krb5kdc/oryx_127.0.98.60.keytab.
I20260926 01:10:20.629575 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:43103
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45453,127.0.98.61:42317,127.0.98.60:43103
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:10:20.713722 7142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:10:20.714020 7142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:10:20.714044 7142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:10:20.715252 7142 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:10:20.715291 7142 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:10:20.715303 7142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:10:20.715314 7142 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:10:20.715325 7142 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:10:20.716853 7142 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45453,127.0.98.61:42317,127.0.98.60:43103
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:43103
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7142
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:10:20.717074 7142 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:10:20.717279 7142 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:10:20.722337 7147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:10:20.722361 7150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:10:20.722642 7142 server_base.cc:1034] running on GCE node
W20260926 01:10:20.722376 7148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:10:20.723614 7142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:10:20.723922 7142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:10:20.725100 7142 hybrid_clock.cc:648] HybridClock initialized: now 1790385020725083 us; error 38 us; skew 500 ppm
Sep 26 01:10:20 dist-test-slave-7b22 krb5kdc[6446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385020, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:10:20.727882 7142 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260926 01:10:20.728394 7142 webserver.cc:460] Webserver started at http://127.0.98.60:37061/ using document root <none> and password file <none>
I20260926 01:10:20.728622 7142 fs_manager.cc:359] Metadata directory not provided
I20260926 01:10:20.728684 7142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:10:20.728801 7142 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:10:20.729971 7142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/master-2/data/instance:
uuid: "3b48fee3b6474a23a70b62be3d2b7f2f"
format_stamp: "Formatted at 2026-09-26 01:10:20 on dist-test-slave-7b22"
server_key: "e0028d673855f170429b9350a02f394c"
server_key_iv: "f513f3e698babe4fbd7e5c87294a410a"
server_key_version: "encryptionkey@0"
I20260926 01:10:20.730351 7142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/master-2/wal/instance:
uuid: "3b48fee3b6474a23a70b62be3d2b7f2f"
format_stamp: "Formatted at 2026-09-26 01:10:20 on dist-test-slave-7b22"
server_key: "e0028d673855f170429b9350a02f394c"
server_key_iv: "f513f3e698babe4fbd7e5c87294a410a"
server_key_version: "encryptionkey@0"
I20260926 01:10:20.731738 7142 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:10:20.732771 7157 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:20.733047 7142 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:10:20.733151 7142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/master-2/wal
uuid: "3b48fee3b6474a23a70b62be3d2b7f2f"
format_stamp: "Formatted at 2026-09-26 01:10:20 on dist-test-slave-7b22"
server_key: "e0028d673855f170429b9350a02f394c"
server_key_iv: "f513f3e698babe4fbd7e5c87294a410a"
server_key_version: "encryptionkey@0"
I20260926 01:10:20.733229 7142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:10:20.745633 7142 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:10:20.748041 7142 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:10:20.748193 7142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:10:20.752300 7142 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:43103
I20260926 01:10:20.752331 7209 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:43103 every 8 connection(s)
I20260926 01:10:20.752717 7142 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/master-2/data/info.pb
I20260926 01:10:20.753907 7210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:10:20.754590 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 7142
I20260926 01:10:20.755645 7210 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45453 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 01:10:20 dist-test-slave-7b22 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385020, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 26 01:10:20 dist-test-slave-7b22 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385019, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260926 01:10:20.769361 6544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:20.756796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39574 (local address 127.0.98.62:45453)
0926 01:10:20.756841 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:20.756843 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:20.756938 (+ 95us) server_negotiation.cc:407] Connection header received
0926 01:10:20.756969 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:20.756970 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:20.756980 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:20.757027 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:20.757474 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:20.757993 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:20.758831 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:20.759032 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:20.759089 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:20.766593 (+ 7504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:10:20.766595 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:10:20.766595 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:10:20.766606 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:10:20.767971 (+ 1365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:20.768514 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:20.768515 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:20.768515 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:20.768568 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:20.768798 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:20.768798 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:20.768798 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:20.768911 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:10:20.768970 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:20.769218 (+ 248us) server_negotiation.cc:299] Negotiation successful
0926 01:10:20.769283 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 01:10:20.769528 7213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:20.756647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45453 (local address 127.0.0.1:39574)
0926 01:10:20.756817 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:20.756858 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:20.756925 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:20.757057 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:20.757060 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:20.757296 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:10:20.757392 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:20.757396 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:20.758073 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:20.758074 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:20.758753 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:20.758754 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:20.759102 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:20.759103 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:20.759166 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:20.764776 (+ 5610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:10:20.764787 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:10:20.766480 (+ 1693us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:10:20.768050 (+ 1570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:10:20.768058 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:10:20.768061 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:10:20.768432 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:10:20.768622 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:10:20.768623 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:10:20.768623 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:10:20.768741 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:10:20.768968 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:10:20.768972 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:10:20.769097 (+ 125us) client_negotiation.cc:769] Sending connection context
0926 01:10:20.769176 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 01:10:20.769337 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":110,"threads_started":1}
I20260926 01:10:20.770077 7212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:20.755383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49086 (local address 127.0.98.60:43103)
0926 01:10:20.755559 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:20.755569 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:20.755600 (+ 31us) server_negotiation.cc:407] Connection header received
0926 01:10:20.756178 (+ 578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:20.756189 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:20.756254 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:20.756406 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:20.757096 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:20.758038 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:20.758695 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:20.758999 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:20.759090 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:20.767134 (+ 8044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:10:20.767143 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:10:20.767144 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:10:20.767151 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:10:20.768904 (+ 1753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:20.769223 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:20.769226 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:20.769226 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:20.769280 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:20.769488 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:20.769489 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:20.769489 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:20.769665 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:10:20.769789 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:20.769911 (+ 122us) server_negotiation.cc:299] Negotiation successful
0926 01:10:20.769983 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":73,"threads_started":1}
I20260926 01:10:20.770184 7210 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42317 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 01:10:20 dist-test-slave-7b22 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385020, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260926 01:10:20.776023 7213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:20.770420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42317 (local address 127.0.0.1:32816)
0926 01:10:20.770534 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:20.770545 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:20.770592 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:20.770710 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:20.770711 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:20.770815 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:10:20.770860 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:20.770861 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:20.771479 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:20.771480 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:20.772032 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:20.772034 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:20.772388 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:20.772388 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:20.772434 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:20.772768 (+ 334us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:10:20.772772 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:10:20.773957 (+ 1185us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:10:20.775305 (+ 1348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:10:20.775306 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:10:20.775306 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:10:20.775468 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:10:20.775598 (+ 130us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:10:20.775598 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:10:20.775599 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:10:20.775638 (+ 39us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:10:20.775805 (+ 167us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:10:20.775805 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:10:20.775838 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 01:10:20.775883 (+ 45us) client_negotiation.cc:240] Negotiation successful
0926 01:10:20.775954 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97}
I20260926 01:10:20.776032 7139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:20.770489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32816 (local address 127.0.98.61:42317)
0926 01:10:20.770545 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:20.770547 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:20.770582 (+ 35us) server_negotiation.cc:407] Connection header received
0926 01:10:20.770640 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:20.770641 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:20.770655 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:20.770698 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:20.770917 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:20.771411 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:20.772115 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:20.772334 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:20.772384 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:20.774023 (+ 1639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:10:20.774025 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:10:20.774025 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:10:20.774032 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:10:20.775239 (+ 1207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:20.775516 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:20.775517 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:20.775517 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:20.775563 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:20.775668 (+ 105us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:20.775668 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:20.775668 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:20.775762 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:10:20.775803 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:20.775908 (+ 105us) server_negotiation.cc:299] Negotiation successful
0926 01:10:20.775971 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 01:10:20.776408 7210 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43103 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 01:10:20 dist-test-slave-7b22 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385020, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260926 01:10:20.782629 7213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:20.776617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43103 (local address 127.0.0.1:49096)
0926 01:10:20.776663 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:20.776673 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:20.776714 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:20.776802 (+ 88us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:20.776803 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:20.776906 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:10:20.776938 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:20.776940 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:20.777489 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:20.777490 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:20.778015 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:20.778017 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:20.778320 (+ 303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:20.778320 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:20.778355 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:20.778704 (+ 349us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:10:20.778707 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:10:20.779914 (+ 1207us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:10:20.781428 (+ 1514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:10:20.781429 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:10:20.781429 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:10:20.781590 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:10:20.781739 (+ 149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:10:20.781740 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:10:20.781740 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:10:20.781972 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:10:20.782313 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:10:20.782314 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:10:20.782355 (+ 41us) client_negotiation.cc:769] Sending connection context
0926 01:10:20.782423 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 01:10:20.782513 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260926 01:10:20.782634 7212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:20.776648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49096 (local address 127.0.98.60:43103)
0926 01:10:20.776678 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:20.776680 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:20.776701 (+ 21us) server_negotiation.cc:407] Connection header received
0926 01:10:20.776750 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:20.776750 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:20.776758 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:20.776800 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:20.776984 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:20.777438 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:20.778079 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:20.778269 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:20.778314 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:20.780042 (+ 1728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:10:20.780044 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:10:20.780044 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:10:20.780054 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:10:20.781324 (+ 1270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:20.781651 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:20.781652 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:20.781652 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:20.781695 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:20.782106 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:20.782107 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:20.782107 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:20.782226 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:10:20.782279 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:20.782441 (+ 162us) server_negotiation.cc:299] Negotiation successful
0926 01:10:20.782560 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":57,"server-negotiator.queue_time_us":14}
I20260926 01:10:20.784683 7210 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b48fee3b6474a23a70b62be3d2b7f2f: Bootstrap starting.
I20260926 01:10:20.785487 7210 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3b48fee3b6474a23a70b62be3d2b7f2f: Neither blocks nor log segments found. Creating new log.
I20260926 01:10:20.785753 7210 log.cc:824] T 00000000000000000000000000000000 P 3b48fee3b6474a23a70b62be3d2b7f2f: Log is configured to *not* fsync() on all Append() calls
I20260926 01:10:20.786621 7210 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b48fee3b6474a23a70b62be3d2b7f2f: No bootstrap required, opened a new log
I20260926 01:10:20.788096 7210 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3b48fee3b6474a23a70b62be3d2b7f2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5afbc257ec7449b189377b1ae0cbebd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45453 } } peers { permanent_uuid: "8971f3cf560c49ae9dd58727c3b669c4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42317 } } peers { permanent_uuid: "3b48fee3b6474a23a70b62be3d2b7f2f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43103 } }
I20260926 01:10:20.788223 7210 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3b48fee3b6474a23a70b62be3d2b7f2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:10:20.788252 7210 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3b48fee3b6474a23a70b62be3d2b7f2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b48fee3b6474a23a70b62be3d2b7f2f, State: Initialized, Role: FOLLOWER
I20260926 01:10:20.788375 7210 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b48fee3b6474a23a70b62be3d2b7f2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5afbc257ec7449b189377b1ae0cbebd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45453 } } peers { permanent_uuid: "8971f3cf560c49ae9dd58727c3b669c4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42317 } } peers { permanent_uuid: "3b48fee3b6474a23a70b62be3d2b7f2f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43103 } }
I20260926 01:10:20.788625 7215 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b48fee3b6474a23a70b62be3d2b7f2f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5afbc257ec7449b189377b1ae0cbebd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45453 } } peers { permanent_uuid: "8971f3cf560c49ae9dd58727c3b669c4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42317 } } peers { permanent_uuid: "3b48fee3b6474a23a70b62be3d2b7f2f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43103 } } }
I20260926 01:10:20.788712 7210 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3b48fee3b6474a23a70b62be3d2b7f2f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:10:20.788720 7215 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b48fee3b6474a23a70b62be3d2b7f2f [sys.catalog]: This master's current role is: FOLLOWER
W20260926 01:10:20.790385 7226 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3b48fee3b6474a23a70b62be3d2b7f2f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 01:10:20.790446 7226 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 01:10:20.792661 7142 master_runner.cc:428] Detected that this master 3b48fee3b6474a23a70b62be3d2b7f2f is joining an existing cluster
I20260926 01:10:20.792714 7142 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.60:43103
I20260926 01:10:20.800189 7212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:20.794034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49102 (local address 127.0.98.60:43103)
0926 01:10:20.794076 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:20.794078 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:20.794266 (+ 188us) server_negotiation.cc:407] Connection header received
0926 01:10:20.794581 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:20.794582 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:20.794589 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:20.794615 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:20.795115 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:20.795550 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:20.796243 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:20.796434 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:20.796487 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:20.797443 (+ 956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:10:20.797444 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:10:20.797445 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:10:20.797451 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:10:20.799046 (+ 1595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:20.799472 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:20.799473 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:20.799473 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:20.799523 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:20.799705 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:20.799705 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:20.799706 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:20.799857 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:10:20.799908 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:20.800056 (+ 148us) server_negotiation.cc:299] Negotiation successful
0926 01:10:20.800124 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 01:10:20.800493 7234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:20.793813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43103 (local address 127.0.0.1:49102)
0926 01:10:20.794183 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:20.794192 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:20.794504 (+ 312us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:20.794683 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:20.794684 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:20.794898 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:10:20.794948 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:20.794951 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:20.795617 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:20.795618 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:20.796175 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:20.796177 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:20.796499 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:20.796500 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:20.796552 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:20.797046 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:10:20.797049 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:10:20.797366 (+ 317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:10:20.799133 (+ 1767us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:10:20.799134 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:10:20.799134 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:10:20.799405 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:10:20.799583 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:10:20.799584 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:10:20.799584 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:10:20.799654 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:10:20.799915 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:10:20.799916 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:10:20.799962 (+ 46us) client_negotiation.cc:769] Sending connection context
0926 01:10:20.800033 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 01:10:20.800189 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"mutex_wait_us":77,"spinlock_wait_cycles":10496,"thread_start_us":95,"threads_started":1}
I20260926 01:10:20.802695 7215 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3b48fee3b6474a23a70b62be3d2b7f2f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:10:20.802855 7215 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3b48fee3b6474a23a70b62be3d2b7f2f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5afbc257ec7449b189377b1ae0cbebd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45453 } } peers { permanent_uuid: "8971f3cf560c49ae9dd58727c3b669c4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42317 } } peers { permanent_uuid: "3b48fee3b6474a23a70b62be3d2b7f2f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43103 } }
I20260926 01:10:20.803072 7215 leader_election.cc:288] T 00000000000000000000000000000000 P 3b48fee3b6474a23a70b62be3d2b7f2f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5afbc257ec7449b189377b1ae0cbebd5 (127.0.98.62:45453), 8971f3cf560c49ae9dd58727c3b669c4 (127.0.98.61:42317)
I20260926 01:10:20.803292 6501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3b48fee3b6474a23a70b62be3d2b7f2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5afbc257ec7449b189377b1ae0cbebd5" is_pre_election: true
I20260926 01:10:20.803339 7098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3b48fee3b6474a23a70b62be3d2b7f2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8971f3cf560c49ae9dd58727c3b669c4" is_pre_election: true
I20260926 01:10:20.803447 7098 raft_consensus.cc:1799] T 00000000000000000000000000000000 P 8971f3cf560c49ae9dd58727c3b669c4 [term 1 LEARNER]: Handling vote request from an unknown peer 3b48fee3b6474a23a70b62be3d2b7f2f
I20260926 01:10:20.803494 7098 raft_consensus.cc:2396] T 00000000000000000000000000000000 P 8971f3cf560c49ae9dd58727c3b669c4 [term 1 LEARNER]: Leader pre-election vote request: Denying vote to candidate 3b48fee3b6474a23a70b62be3d2b7f2f for term 1 because replica has last-logged OpId of term: 1 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260926 01:10:20.803686 7160 leader_election.cc:302] T 00000000000000000000000000000000 P 3b48fee3b6474a23a70b62be3d2b7f2f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b48fee3b6474a23a70b62be3d2b7f2f; no voters: 5afbc257ec7449b189377b1ae0cbebd5, 8971f3cf560c49ae9dd58727c3b669c4
I20260926 01:10:20.803823 7215 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3b48fee3b6474a23a70b62be3d2b7f2f [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:10:20.804770 7215 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 3b48fee3b6474a23a70b62be3d2b7f2f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:10:20.821388 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 6553
I20260926 01:10:20.822749 6544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:20.793884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39576 (local address 127.0.98.62:45453)
0926 01:10:20.794213 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:20.794215 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:20.794224 (+ 9us) server_negotiation.cc:407] Connection header received
0926 01:10:20.794250 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:20.794250 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:20.794261 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:20.794323 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:20.794733 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:20.795514 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:20.796621 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:20.796854 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:20.796910 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:20.819618 (+ 22708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:10:20.819621 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:10:20.819621 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:10:20.819631 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:10:20.821415 (+ 1784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:20.821776 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:20.821778 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:20.821778 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:20.821870 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:20.822060 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:20.822060 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:20.822060 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:20.822291 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:10:20.822349 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:20.822559 (+ 210us) server_negotiation.cc:299] Negotiation successful
0926 01:10:20.822642 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272}
I20260926 01:10:20.823094 7232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:20.793811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45453 (local address 127.0.0.1:39576)
0926 01:10:20.793983 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:20.793995 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:20.794048 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:20.794331 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:20.794332 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:20.794544 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:10:20.794630 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:20.794632 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:20.795606 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:20.795607 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:20.796539 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:20.796541 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:20.796914 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:20.796914 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:20.796973 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:20.819011 (+ 22038us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:10:20.819017 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:10:20.819489 (+ 472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:10:20.821500 (+ 2011us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:10:20.821501 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:10:20.821501 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:10:20.821700 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:10:20.821941 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:10:20.821942 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:10:20.821942 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:10:20.821999 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:10:20.822401 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:10:20.822403 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:10:20.822450 (+ 47us) client_negotiation.cc:769] Sending connection context
0926 01:10:20.822874 (+ 424us) client_negotiation.cc:240] Negotiation successful
0926 01:10:20.822986 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":96,"threads_started":1}
I20260926 01:10:20.825353 7139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:20.794411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32826 (local address 127.0.98.61:42317)
0926 01:10:20.794448 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:20.794449 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:20.794557 (+ 108us) server_negotiation.cc:407] Connection header received
0926 01:10:20.794639 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:20.794640 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:20.794649 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:20.794677 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:20.818344 (+ 23667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:20.818856 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:20.819828 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:20.820072 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:20.820134 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:20.821424 (+ 1290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:10:20.821426 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:10:20.821426 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:10:20.821434 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:10:20.824171 (+ 2737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:20.824553 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:20.824554 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:20.824554 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:20.824610 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:20.824801 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:20.824802 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:20.824802 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:20.824940 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:10:20.824981 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:20.825169 (+ 188us) server_negotiation.cc:299] Negotiation successful
0926 01:10:20.825255 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 01:10:20.825888 7233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:20.794071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42317 (local address 127.0.0.1:32826)
0926 01:10:20.794510 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:20.794519 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:20.794577 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:20.794731 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:20.794731 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:20.818127 (+ 23396us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:10:20.818223 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:20.818228 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:20.818954 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:20.818955 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:20.819749 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:20.819751 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:20.820485 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:20.820487 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:20.820565 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:20.820942 (+ 377us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:10:20.820947 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:10:20.821319 (+ 372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:10:20.824264 (+ 2945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:10:20.824265 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:10:20.824266 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:10:20.824471 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:10:20.824686 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:10:20.824686 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:10:20.824687 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:10:20.824739 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:10:20.825024 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:10:20.825025 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:10:20.825068 (+ 43us) client_negotiation.cc:769] Sending connection context
0926 01:10:20.825626 (+ 558us) client_negotiation.cc:240] Negotiation successful
0926 01:10:20.825740 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":61,"threads_started":1}
I20260926 01:10:20.827721 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:35643
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:45453
--webserver_port=43807
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:45453,127.0.98.61:42317
--tserver_master_addrs=127.0.98.62:45453,127.0.98.61:42317,127.0.98.60:43103 with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 01:10:20 dist-test-slave-7b22 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385019, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260926 01:10:20.833619 7212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:20.826410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49104 (local address 127.0.98.60:43103)
0926 01:10:20.826453 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:20.826454 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:20.826460 (+ 6us) server_negotiation.cc:407] Connection header received
0926 01:10:20.826491 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:20.826492 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:20.826501 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:20.826540 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:20.826876 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:20.827511 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:20.828399 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:20.828621 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:20.828678 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:20.830758 (+ 2080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:10:20.830759 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:10:20.830760 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:10:20.830768 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:10:20.832345 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:20.832822 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:20.832823 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:20.832824 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:20.832888 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:20.833089 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:20.833090 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:20.833090 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:20.833226 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:10:20.833288 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:20.833436 (+ 148us) server_negotiation.cc:299] Negotiation successful
0926 01:10:20.833517 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260926 01:10:20.833622 7048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:20.826213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43103 (local address 127.0.0.1:49104)
0926 01:10:20.826253 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:20.826265 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:20.826310 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:20.826583 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:20.826584 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:20.826752 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:10:20.826802 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:20.826804 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:20.827591 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:20.827592 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:20.828322 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:20.828325 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:20.828692 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:20.828693 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:20.828760 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:20.829159 (+ 399us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:10:20.829164 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:10:20.830669 (+ 1505us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:10:20.832463 (+ 1794us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:10:20.832464 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:10:20.832465 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:10:20.832746 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:10:20.832962 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:10:20.832964 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:10:20.832964 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:10:20.833029 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:10:20.833285 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:10:20.833287 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:10:20.833338 (+ 51us) client_negotiation.cc:769] Sending connection context
0926 01:10:20.833409 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 01:10:20.833521 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18}
I20260926 01:10:20.834434 6491 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.60:43103: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "3b48fee3b6474a23a70b62be3d2b7f2f"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.60"
port: 43103
}
attrs {
promote: true
}
}
dest_uuid: "5afbc257ec7449b189377b1ae0cbebd5"
cas_config_opid_index: 5
I20260926 01:10:20.834713 6491 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5afbc257ec7449b189377b1ae0cbebd5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5afbc257ec7449b189377b1ae0cbebd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45453 } } peers { permanent_uuid: "8971f3cf560c49ae9dd58727c3b669c4" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 42317 } attrs { promote: true } } peers { permanent_uuid: "3b48fee3b6474a23a70b62be3d2b7f2f" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43103 } attrs { promote: true } }
I20260926 01:10:20.835081 7050 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5afbc257ec7449b189377b1ae0cbebd5 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 3b48fee3b6474a23a70b62be3d2b7f2f (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5afbc257ec7449b189377b1ae0cbebd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45453 } } peers { permanent_uuid: "8971f3cf560c49ae9dd58727c3b669c4" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 42317 } attrs { promote: true } } peers { permanent_uuid: "3b48fee3b6474a23a70b62be3d2b7f2f" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43103 } attrs { promote: true } } }
I20260926 01:10:20.836087 7050 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.60:43103
I20260926 01:10:20.836165 7050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5afbc257ec7449b189377b1ae0cbebd5 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "5afbc257ec7449b189377b1ae0cbebd5" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5afbc257ec7449b189377b1ae0cbebd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45453 } } peers { permanent_uuid: "8971f3cf560c49ae9dd58727c3b669c4" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 42317 } attrs { promote: true } } peers { permanent_uuid: "3b48fee3b6474a23a70b62be3d2b7f2f" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43103 } attrs { promote: true } } }
I20260926 01:10:20.836243 7050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5afbc257ec7449b189377b1ae0cbebd5 [sys.catalog]: This master's current role is: LEADER
I20260926 01:10:20.836509 7184 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3b48fee3b6474a23a70b62be3d2b7f2f [term 1 FOLLOWER]: Refusing update from remote peer 5afbc257ec7449b189377b1ae0cbebd5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260926 01:10:20.836942 7049 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5afbc257ec7449b189377b1ae0cbebd5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b48fee3b6474a23a70b62be3d2b7f2f" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43103 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:10:20.838979 7048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:20.835131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42317 (local address 127.0.0.1:32838)
0926 01:10:20.835163 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:20.835173 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:20.835209 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:20.835401 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:20.835402 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:20.835404 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:10:20.835444 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:20.835446 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:20.836419 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:20.836420 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:20.837643 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:20.837646 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:20.838700 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:20.838701 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:20.838772 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:20.838799 (+ 27us) client_negotiation.cc:769] Sending connection context
0926 01:10:20.838850 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 01:10:20.838905 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16}
I20260926 01:10:20.839131 7184 raft_consensus.cc:858] T 00000000000000000000000000000000 P 3b48fee3b6474a23a70b62be3d2b7f2f [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5afbc257ec7449b189377b1ae0cbebd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45453 } } peers { permanent_uuid: "8971f3cf560c49ae9dd58727c3b669c4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42317 } } peers { permanent_uuid: "3b48fee3b6474a23a70b62be3d2b7f2f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43103 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5afbc257ec7449b189377b1ae0cbebd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45453 } } peers { permanent_uuid: "8971f3cf560c49ae9dd58727c3b669c4" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 42317 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5afbc257ec7449b189377b1ae0cbebd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45453 } } peers { permanent_uuid: "8971f3cf560c49ae9dd58727c3b669c4" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 42317 } attrs { promote: true } } peers { permanent_uuid: "3b48fee3b6474a23a70b62be3d2b7f2f" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43103 } attrs { promote: true } }
W20260926 01:10:20.839197 7184 raft_consensus.cc:1565] T 00000000000000000000000000000000 P 3b48fee3b6474a23a70b62be3d2b7f2f [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5afbc257ec7449b189377b1ae0cbebd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45453 } } peers { permanent_uuid: "8971f3cf560c49ae9dd58727c3b669c4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42317 } } peers { permanent_uuid: "3b48fee3b6474a23a70b62be3d2b7f2f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43103 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5afbc257ec7449b189377b1ae0cbebd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45453 } } peers { permanent_uuid: "8971f3cf560c49ae9dd58727c3b669c4" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 42317 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5afbc257ec7449b189377b1ae0cbebd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45453 } } peers { permanent_uuid: "8971f3cf560c49ae9dd58727c3b669c4" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 42317 } attrs { promote: true } } peers { permanent_uuid: "3b48fee3b6474a23a70b62be3d2b7f2f" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43103 } attrs { promote: true } }
I20260926 01:10:20.839222 7139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:20.835233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32838 (local address 127.0.98.61:42317)
0926 01:10:20.835271 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:20.835272 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:20.835279 (+ 7us) server_negotiation.cc:407] Connection header received
0926 01:10:20.835301 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:20.835302 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:20.835324 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:20.835386 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:10:20.835514 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:20.836302 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:20.837797 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:20.838486 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:20.838547 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:20.838622 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:20.839115 (+ 493us) server_negotiation.cc:299] Negotiation successful
0926 01:10:20.839157 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 01:10:20.839401 7098 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8971f3cf560c49ae9dd58727c3b669c4 [term 1 LEARNER]: Refusing update from remote peer 5afbc257ec7449b189377b1ae0cbebd5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260926 01:10:20.839615 7049 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5afbc257ec7449b189377b1ae0cbebd5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8971f3cf560c49ae9dd58727c3b669c4" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 42317 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260926 01:10:20.839700 7184 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3b48fee3b6474a23a70b62be3d2b7f2f [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 8971f3cf560c49ae9dd58727c3b669c4 (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5afbc257ec7449b189377b1ae0cbebd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45453 } } peers { permanent_uuid: "8971f3cf560c49ae9dd58727c3b669c4" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 42317 } attrs { promote: true } } }
I20260926 01:10:20.840128 7098 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8971f3cf560c49ae9dd58727c3b669c4 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 3b48fee3b6474a23a70b62be3d2b7f2f (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5afbc257ec7449b189377b1ae0cbebd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45453 } } peers { permanent_uuid: "8971f3cf560c49ae9dd58727c3b669c4" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 42317 } attrs { promote: true } } peers { permanent_uuid: "3b48fee3b6474a23a70b62be3d2b7f2f" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43103 } attrs { promote: true } } }
I20260926 01:10:20.840884 7215 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b48fee3b6474a23a70b62be3d2b7f2f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5afbc257ec7449b189377b1ae0cbebd5. Latest consensus state: current_term: 1 leader_uuid: "5afbc257ec7449b189377b1ae0cbebd5" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5afbc257ec7449b189377b1ae0cbebd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45453 } } peers { permanent_uuid: "8971f3cf560c49ae9dd58727c3b669c4" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 42317 } attrs { promote: true } } }
I20260926 01:10:20.840968 7215 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b48fee3b6474a23a70b62be3d2b7f2f [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260926 01:10:20.840999 7215 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b48fee3b6474a23a70b62be3d2b7f2f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5afbc257ec7449b189377b1ae0cbebd5" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5afbc257ec7449b189377b1ae0cbebd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45453 } } peers { permanent_uuid: "8971f3cf560c49ae9dd58727c3b669c4" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 42317 } attrs { promote: true } } }
I20260926 01:10:20.841038 7215 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b48fee3b6474a23a70b62be3d2b7f2f [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260926 01:10:20.841059 7215 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b48fee3b6474a23a70b62be3d2b7f2f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5afbc257ec7449b189377b1ae0cbebd5" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5afbc257ec7449b189377b1ae0cbebd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45453 } } peers { permanent_uuid: "8971f3cf560c49ae9dd58727c3b669c4" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 42317 } attrs { promote: true } } }
I20260926 01:10:20.841090 7215 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b48fee3b6474a23a70b62be3d2b7f2f [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260926 01:10:20.841156 7127 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8971f3cf560c49ae9dd58727c3b669c4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5afbc257ec7449b189377b1ae0cbebd5. Latest consensus state: current_term: 1 leader_uuid: "5afbc257ec7449b189377b1ae0cbebd5" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5afbc257ec7449b189377b1ae0cbebd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45453 } } peers { permanent_uuid: "8971f3cf560c49ae9dd58727c3b669c4" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 42317 } attrs { promote: true } } peers { permanent_uuid: "3b48fee3b6474a23a70b62be3d2b7f2f" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43103 } attrs { promote: true } } }
I20260926 01:10:20.841233 7127 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8971f3cf560c49ae9dd58727c3b669c4 [sys.catalog]: This master's current role is: LEARNER
I20260926 01:10:20.841261 7127 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8971f3cf560c49ae9dd58727c3b669c4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5afbc257ec7449b189377b1ae0cbebd5" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5afbc257ec7449b189377b1ae0cbebd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45453 } } peers { permanent_uuid: "8971f3cf560c49ae9dd58727c3b669c4" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 42317 } attrs { promote: true } } peers { permanent_uuid: "3b48fee3b6474a23a70b62be3d2b7f2f" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43103 } attrs { promote: true } } }
I20260926 01:10:20.841305 7127 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8971f3cf560c49ae9dd58727c3b669c4 [sys.catalog]: This master's current role is: LEARNER
I20260926 01:10:20.841727 7142 master.cc:537] Master@127.0.98.60:43103 shutting down...
I20260926 01:10:20.841800 7049 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 5afbc257ec7449b189377b1ae0cbebd5: attempting to promote NON_VOTER 8971f3cf560c49ae9dd58727c3b669c4 to VOTER
I20260926 01:10:20.842018 7049 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5afbc257ec7449b189377b1ae0cbebd5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5afbc257ec7449b189377b1ae0cbebd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45453 } } peers { permanent_uuid: "8971f3cf560c49ae9dd58727c3b669c4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42317 } attrs { promote: false } } peers { permanent_uuid: "3b48fee3b6474a23a70b62be3d2b7f2f" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43103 } attrs { promote: true } }
I20260926 01:10:20.842453 7098 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8971f3cf560c49ae9dd58727c3b669c4 [term 1 LEARNER]: Refusing update from remote peer 5afbc257ec7449b189377b1ae0cbebd5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260926 01:10:20.842651 7049 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5afbc257ec7449b189377b1ae0cbebd5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8971f3cf560c49ae9dd58727c3b669c4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42317 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260926 01:10:20.843294 6476 consensus_peers.cc:487] T 00000000000000000000000000000000 P 5afbc257ec7449b189377b1ae0cbebd5 -> Peer 3b48fee3b6474a23a70b62be3d2b7f2f (127.0.98.60:43103): Couldn't send request to peer 3b48fee3b6474a23a70b62be3d2b7f2f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260926 01:10:20.843616 7050 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5afbc257ec7449b189377b1ae0cbebd5 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 8971f3cf560c49ae9dd58727c3b669c4 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5afbc257ec7449b189377b1ae0cbebd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45453 } } peers { permanent_uuid: "8971f3cf560c49ae9dd58727c3b669c4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42317 } attrs { promote: false } } peers { permanent_uuid: "3b48fee3b6474a23a70b62be3d2b7f2f" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43103 } attrs { promote: true } } }
I20260926 01:10:20.844466 7050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5afbc257ec7449b189377b1ae0cbebd5 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "5afbc257ec7449b189377b1ae0cbebd5" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5afbc257ec7449b189377b1ae0cbebd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45453 } } peers { permanent_uuid: "8971f3cf560c49ae9dd58727c3b669c4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42317 } attrs { promote: false } } peers { permanent_uuid: "3b48fee3b6474a23a70b62be3d2b7f2f" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43103 } attrs { promote: true } } }
I20260926 01:10:20.844542 7050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5afbc257ec7449b189377b1ae0cbebd5 [sys.catalog]: This master's current role is: LEADER
I20260926 01:10:20.844594 7050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5afbc257ec7449b189377b1ae0cbebd5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5afbc257ec7449b189377b1ae0cbebd5" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5afbc257ec7449b189377b1ae0cbebd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45453 } } peers { permanent_uuid: "8971f3cf560c49ae9dd58727c3b669c4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42317 } attrs { promote: false } } peers { permanent_uuid: "3b48fee3b6474a23a70b62be3d2b7f2f" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43103 } attrs { promote: true } } }
I20260926 01:10:20.844638 7050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5afbc257ec7449b189377b1ae0cbebd5 [sys.catalog]: This master's current role is: LEADER
I20260926 01:10:20.844847 7098 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8971f3cf560c49ae9dd58727c3b669c4 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 8971f3cf560c49ae9dd58727c3b669c4 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5afbc257ec7449b189377b1ae0cbebd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45453 } } peers { permanent_uuid: "8971f3cf560c49ae9dd58727c3b669c4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42317 } attrs { promote: false } } peers { permanent_uuid: "3b48fee3b6474a23a70b62be3d2b7f2f" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43103 } attrs { promote: true } } }
I20260926 01:10:20.846330 7236 mvcc.cc:205] Tried to move back new op lower bound from 7333417041104015360 to 7333417040994557952. Current Snapshot: MvccSnapshot[applied={T|T < 7333417041003593728}]
I20260926 01:10:20.846555 7241 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8971f3cf560c49ae9dd58727c3b669c4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5afbc257ec7449b189377b1ae0cbebd5" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5afbc257ec7449b189377b1ae0cbebd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45453 } } peers { permanent_uuid: "8971f3cf560c49ae9dd58727c3b669c4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42317 } attrs { promote: false } } peers { permanent_uuid: "3b48fee3b6474a23a70b62be3d2b7f2f" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43103 } attrs { promote: true } } }
I20260926 01:10:20.846720 7241 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8971f3cf560c49ae9dd58727c3b669c4 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 01:10:20.846884 7142 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 3b48fee3b6474a23a70b62be3d2b7f2f [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260926 01:10:20.846988 7142 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 3b48fee3b6474a23a70b62be3d2b7f2f [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260926 01:10:20.847018 7142 tablet_replica.cc:330] T 00000000000000000000000000000000 P 3b48fee3b6474a23a70b62be3d2b7f2f: stopping tablet replica
I20260926 01:10:20.851259 7142 master.cc:552] Master@127.0.98.60:43103 shutdown complete.
I20260926 01:10:20.851311 7142 master_runner.cc:305] Clearing existing system tablet
I20260926 01:10:20.851433 7142 fs_manager.cc:359] Metadata directory not provided
I20260926 01:10:20.851469 7142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:10:20.852492 7142 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:10:20.853068 7247 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:20.853204 7142 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:10:20.853247 7142 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:20.853313 7142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/master-2/wal
uuid: "3b48fee3b6474a23a70b62be3d2b7f2f"
format_stamp: "Formatted at 2026-09-26 01:10:20 on dist-test-slave-7b22"
server_key: "e0028d673855f170429b9350a02f394c"
server_key_iv: "f513f3e698babe4fbd7e5c87294a410a"
server_key_version: "encryptionkey@0"
I20260926 01:10:20.854053 7142 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 3b48fee3b6474a23a70b62be3d2b7f2f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:10:20.855736 7142 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 3b48fee3b6474a23a70b62be3d2b7f2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260926 01:10:20.855809 7142 log.cc:1196] T 00000000000000000000000000000000 P 3b48fee3b6474a23a70b62be3d2b7f2f: Deleting WAL directory at /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260926 01:10:20.856040 7142 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 3b48fee3b6474a23a70b62be3d2b7f2f: Deleting consensus metadata
I20260926 01:10:20.856153 7142 master_runner.cc:315] Copying system tablet from 127.0.98.62:45453
I20260926 01:10:20.856732 7142 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 3b48fee3b6474a23a70b62be3d2b7f2f: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:45453
I20260926 01:10:20.863801 7253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:20.857027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45453 (local address 127.0.0.1:39588)
0926 01:10:20.857159 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:20.857170 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:20.857238 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:20.857390 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:20.857391 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:20.857590 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:10:20.857644 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:20.857645 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:20.858622 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:20.858623 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:20.859443 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:20.859446 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:20.859924 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:20.859925 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:20.859990 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:20.860505 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:10:20.860510 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:10:20.860907 (+ 397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:10:20.862738 (+ 1831us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:10:20.862739 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:10:20.862740 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:10:20.862951 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:10:20.863160 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:10:20.863161 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:10:20.863161 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:10:20.863225 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:10:20.863489 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:10:20.863490 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:10:20.863534 (+ 44us) client_negotiation.cc:769] Sending connection context
0926 01:10:20.863589 (+ 55us) client_negotiation.cc:240] Negotiation successful
0926 01:10:20.863694 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":76,"threads_started":1}
I20260926 01:10:20.864344 6544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:20.857099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39588 (local address 127.0.98.62:45453)
0926 01:10:20.857147 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:20.857149 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:20.857227 (+ 78us) server_negotiation.cc:407] Connection header received
0926 01:10:20.857290 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:20.857291 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:20.857301 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:20.857374 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:20.857729 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:20.858522 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:20.859536 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:20.859844 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:20.859920 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:20.861000 (+ 1080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:10:20.861002 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:10:20.861003 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:10:20.861012 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:10:20.862651 (+ 1639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:20.863037 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:20.863039 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:20.863039 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:20.863100 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:20.863281 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:20.863282 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:20.863282 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:20.863413 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:10:20.863466 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:20.864157 (+ 691us) server_negotiation.cc:299] Negotiation successful
0926 01:10:20.864258 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 01:10:20.864848 6511 tablet_copy_service.cc:140] P 5afbc257ec7449b189377b1ae0cbebd5: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 3b48fee3b6474a23a70b62be3d2b7f2f ({username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:39588)
I20260926 01:10:20.864912 6511 tablet_copy_service.cc:161] P 5afbc257ec7449b189377b1ae0cbebd5: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 3b48fee3b6474a23a70b62be3d2b7f2f at {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:39588: session id = 3b48fee3b6474a23a70b62be3d2b7f2f-00000000000000000000000000000000
I20260926 01:10:20.865537 6511 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 5afbc257ec7449b189377b1ae0cbebd5: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 01:10:20.866118 7142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260926 01:10:20.867477 7142 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 3b48fee3b6474a23a70b62be3d2b7f2f: tablet copy: Starting download of 0 data blocks...
I20260926 01:10:20.867712 7142 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 3b48fee3b6474a23a70b62be3d2b7f2f: tablet copy: Starting download of 1 WAL segments...
I20260926 01:10:20.868613 7142 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 3b48fee3b6474a23a70b62be3d2b7f2f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 01:10:20.869637 6511 tablet_copy_service.cc:342] P 5afbc257ec7449b189377b1ae0cbebd5: Request end of tablet copy session 3b48fee3b6474a23a70b62be3d2b7f2f-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:39588
I20260926 01:10:20.869700 6511 tablet_copy_service.cc:433] P 5afbc257ec7449b189377b1ae0cbebd5: ending tablet copy session 3b48fee3b6474a23a70b62be3d2b7f2f-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 3b48fee3b6474a23a70b62be3d2b7f2f
I20260926 01:10:20.872612 7142 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:10:20.873777 7254 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:10:20.874112 7255 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:10:20.874255 7142 server_base.cc:1034] running on GCE node
W20260926 01:10:20.874414 7257 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:10:20.874579 7142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:10:20.874727 7142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:10:20.875833 7142 hybrid_clock.cc:648] HybridClock initialized: now 1790385020875818 us; error 45 us; skew 500 ppm
Sep 26 01:10:20 dist-test-slave-7b22 krb5kdc[6446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385020, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:10:20.877431 7142 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260926 01:10:20.877799 7142 webserver.cc:460] Webserver started at http://127.0.98.60:44023/ using document root <none> and password file <none>
I20260926 01:10:20.878036 7142 fs_manager.cc:359] Metadata directory not provided
I20260926 01:10:20.878124 7142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:10:20.879000 7142 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 01:10:20.879688 7264 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:20.879840 7142 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:20.879920 7142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/master-2/wal
uuid: "3b48fee3b6474a23a70b62be3d2b7f2f"
format_stamp: "Formatted at 2026-09-26 01:10:20 on dist-test-slave-7b22"
server_key: "e0028d673855f170429b9350a02f394c"
server_key_iv: "f513f3e698babe4fbd7e5c87294a410a"
server_key_version: "encryptionkey@0"
I20260926 01:10:20.880028 7142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:10:20.895985 7142 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:10:20.897784 7142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:10:20.900686 7142 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:43103
I20260926 01:10:20.900729 7316 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:43103 every 8 connection(s)
I20260926 01:10:20.901152 7142 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/master-2/data/info.pb
I20260926 01:10:20.902216 7317 sys_catalog.cc:263] Verifying existing consensus state
I20260926 01:10:20.902707 7317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b48fee3b6474a23a70b62be3d2b7f2f: Bootstrap starting.
I20260926 01:10:20.904928 7317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b48fee3b6474a23a70b62be3d2b7f2f: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 01:10:20.905122 7317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b48fee3b6474a23a70b62be3d2b7f2f: Bootstrap complete.
I20260926 01:10:20.905308 7317 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3b48fee3b6474a23a70b62be3d2b7f2f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5afbc257ec7449b189377b1ae0cbebd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45453 } } peers { permanent_uuid: "8971f3cf560c49ae9dd58727c3b669c4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42317 } attrs { promote: false } } peers { permanent_uuid: "3b48fee3b6474a23a70b62be3d2b7f2f" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43103 } attrs { promote: true } }
I20260926 01:10:20.905393 7317 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3b48fee3b6474a23a70b62be3d2b7f2f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3b48fee3b6474a23a70b62be3d2b7f2f, State: Initialized, Role: LEARNER
I20260926 01:10:20.905427 7317 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b48fee3b6474a23a70b62be3d2b7f2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5afbc257ec7449b189377b1ae0cbebd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45453 } } peers { permanent_uuid: "8971f3cf560c49ae9dd58727c3b669c4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42317 } attrs { promote: false } } peers { permanent_uuid: "3b48fee3b6474a23a70b62be3d2b7f2f" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43103 } attrs { promote: true } }
I20260926 01:10:20.905580 7317 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3b48fee3b6474a23a70b62be3d2b7f2f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:10:20.905906 7320 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b48fee3b6474a23a70b62be3d2b7f2f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5afbc257ec7449b189377b1ae0cbebd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45453 } } peers { permanent_uuid: "8971f3cf560c49ae9dd58727c3b669c4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42317 } attrs { promote: false } } peers { permanent_uuid: "3b48fee3b6474a23a70b62be3d2b7f2f" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43103 } attrs { promote: true } } }
I20260926 01:10:20.906081 7320 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b48fee3b6474a23a70b62be3d2b7f2f [sys.catalog]: This master's current role is: LEARNER
I20260926 01:10:20.907083 7331 catalog_manager.cc:1259] Loaded cluster ID: d16d612222b24a05bd8871dd9f254e18
I20260926 01:10:20.907171 7331 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3b48fee3b6474a23a70b62be3d2b7f2f: loading cluster ID for follower catalog manager: success
I20260926 01:10:20.908905 7331 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3b48fee3b6474a23a70b62be3d2b7f2f: acquiring CA information for follower catalog manager: success
I20260926 01:10:20.909302 7331 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3b48fee3b6474a23a70b62be3d2b7f2f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260926 01:10:20.932410 7235 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:10:20.932667 7235 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:10:20.932701 7235 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:10:20.934000 7235 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:10:20.934047 7235 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:10:20.934077 7235 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 01:10:20.935699 7235 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:35643
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=43807
--tserver_master_addrs=127.0.98.62:45453,127.0.98.61:42317,127.0.98.60:43103
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7235
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:10:20.935945 7235 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:10:20.936165 7235 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:10:20.941462 7337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:10:20.941587 7235 server_base.cc:1034] running on GCE node
W20260926 01:10:20.941711 7336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:10:20.941596 7339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:10:20.942004 7235 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:10:20.942233 7235 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:10:20.943393 7235 hybrid_clock.cc:648] HybridClock initialized: now 1790385020943378 us; error 35 us; skew 500 ppm
Sep 26 01:10:20 dist-test-slave-7b22 krb5kdc[6446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385020, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:10:20.946537 7235 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260926 01:10:20.946982 7235 webserver.cc:460] Webserver started at http://127.0.98.1:43807/ using document root <none> and password file <none>
I20260926 01:10:20.947197 7235 fs_manager.cc:359] Metadata directory not provided
I20260926 01:10:20.947259 7235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:10:20.948665 7235 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:10:20.949330 7346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:20.949510 7235 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:10:20.949607 7235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/ts-0/wal
uuid: "2264518cb0fe444da3e1606269628249"
format_stamp: "Formatted at 2026-09-26 01:10:19 on dist-test-slave-7b22"
server_key: "48d221961e9055483bd294f110dc6b34"
server_key_iv: "a6c14ae5682365af06cfad225051ae2c"
server_key_version: "encryptionkey@0"
I20260926 01:10:20.950006 7235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:10:20.972483 7235 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:10:20.974846 7235 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:10:20.974974 7235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:10:20.975189 7235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:10:20.975528 7235 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:10:20.975577 7235 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:20.975606 7235 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:10:20.975623 7235 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:20.981627 7235 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35643
I20260926 01:10:20.981670 7459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35643 every 8 connection(s)
I20260926 01:10:20.982041 7235 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/ts-0/data/info.pb
I20260926 01:10:20.983445 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 7235
I20260926 01:10:20.983585 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 6687
I20260926 01:10:20.990707 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:39297
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:45453
--webserver_port=38159
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:45453,127.0.98.61:42317
--tserver_master_addrs=127.0.98.62:45453,127.0.98.61:42317,127.0.98.60:43103 with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 01:10:20 dist-test-slave-7b22 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385020, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 26 01:10:20 dist-test-slave-7b22 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385020, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 26 01:10:20 dist-test-slave-7b22 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385020, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 01:10:20.997079 7464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:20.982710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40541 (local address 127.0.98.60:43103)
0926 01:10:20.987053 (+ 4343us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:20.987055 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:20.987063 (+ 8us) server_negotiation.cc:407] Connection header received
0926 01:10:20.987086 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:20.987086 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:20.987096 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:20.988163 (+ 1067us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:20.988207 (+ 44us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:20.989251 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:20.991395 (+ 2144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:20.991685 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:20.992020 (+ 335us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:20.994206 (+ 2186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:10:20.994208 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:10:20.994208 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:10:20.994216 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:10:20.995848 (+ 1632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:20.996228 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:20.996229 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:20.996230 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:20.996290 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:20.996471 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:20.996472 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:20.996472 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:20.996603 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:10:20.996657 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:20.996892 (+ 235us) server_negotiation.cc:299] Negotiation successful
0926 01:10:20.996977 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4323,"thread_start_us":76,"threads_started":1}
I20260926 01:10:20.997169 7467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:20.983037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42317 (local address 127.0.98.1:37201)
0926 01:10:20.983297 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:20.983307 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:20.988471 (+ 5164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:20.988676 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:20.988677 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:20.988825 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:10:20.988881 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:20.988882 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:20.989770 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:20.989771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:20.990724 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:20.990727 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:20.991240 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:20.991241 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:20.991385 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:20.991973 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:10:20.991982 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:10:20.993649 (+ 1667us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:10:20.995461 (+ 1812us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:10:20.995469 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:10:20.995471 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:10:20.995762 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:10:20.995974 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:10:20.995975 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:10:20.995975 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:10:20.996111 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:10:20.996439 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:10:20.996445 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:10:20.996618 (+ 173us) client_negotiation.cc:769] Sending connection context
0926 01:10:20.996727 (+ 109us) client_negotiation.cc:240] Negotiation successful
0926 01:10:20.996914 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"mutex_wait_us":7,"thread_start_us":60,"threads_started":1}
I20260926 01:10:20.997859 7139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:20.983142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37201 (local address 127.0.98.61:42317)
0926 01:10:20.983195 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:20.983197 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:20.983363 (+ 166us) server_negotiation.cc:407] Connection header received
0926 01:10:20.988571 (+ 5208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:20.988572 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:20.988585 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:20.988615 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:20.988970 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:20.989681 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:20.990822 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:20.991121 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:20.991159 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:20.993758 (+ 2599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:10:20.993760 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:10:20.993760 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:10:20.993769 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:10:20.995384 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:20.995844 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:20.995845 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:20.995845 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:20.995905 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:20.996180 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:20.996181 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:20.996181 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:20.996339 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:10:20.996411 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:20.997669 (+ 1258us) server_negotiation.cc:299] Negotiation successful
0926 01:10:20.997748 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 01:10:20.998049 7466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:20.983194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45453 (local address 127.0.98.1:46683)
0926 01:10:20.986928 (+ 3734us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:20.986937 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:20.986979 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:20.988288 (+ 1309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:20.988289 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:20.988502 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:10:20.988568 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:20.988570 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:20.989522 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:20.989523 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:20.991307 (+ 1784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:20.991309 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:20.991805 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:20.991806 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:20.991883 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:20.992330 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:10:20.992334 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:10:20.995269 (+ 2935us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:10:20.996976 (+ 1707us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:10:20.996976 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:10:20.996977 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:10:20.997156 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:10:20.997377 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:10:20.997377 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:10:20.997378 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:10:20.997431 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:10:20.997733 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:10:20.997734 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:10:20.997774 (+ 40us) client_negotiation.cc:769] Sending connection context
0926 01:10:20.997865 (+ 91us) client_negotiation.cc:240] Negotiation successful
0926 01:10:20.997959 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3710,"mutex_wait_us":2,"thread_start_us":57,"threads_started":1}
I20260926 01:10:20.998080 6544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:20.983253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46683 (local address 127.0.98.62:45453)
0926 01:10:20.983301 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:20.983303 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:20.988099 (+ 4796us) server_negotiation.cc:407] Connection header received
0926 01:10:20.988139 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:20.988140 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:20.988154 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:20.988229 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:20.988652 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:20.989412 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:20.991391 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:20.991676 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:20.991743 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:20.995378 (+ 3635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:10:20.995380 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:10:20.995380 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:10:20.995388 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:10:20.996893 (+ 1505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:20.997249 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:20.997250 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:20.997250 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:20.997311 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:20.997514 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:20.997516 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:20.997516 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:20.997660 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:10:20.997730 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:20.997911 (+ 181us) server_negotiation.cc:299] Negotiation successful
0926 01:10:20.997992 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 01:10:20.998221 7462 heartbeater.cc:347] Connected to a master server at 127.0.98.61:42317
I20260926 01:10:20.998338 7462 heartbeater.cc:464] Registering TS with master...
I20260926 01:10:20.998420 7461 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45453
I20260926 01:10:20.998461 7461 heartbeater.cc:464] Registering TS with master...
I20260926 01:10:20.998543 7462 heartbeater.cc:511] Master 127.0.98.61:42317 requested a full tablet report, sending...
I20260926 01:10:20.998543 7461 heartbeater.cc:511] Master 127.0.98.62:45453 requested a full tablet report, sending...
I20260926 01:10:20.998993 7465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:20.982700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43103 (local address 127.0.98.1:40541)
0926 01:10:20.982862 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:20.982908 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:20.986709 (+ 3801us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:20.987232 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:20.987236 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:20.987860 (+ 624us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:10:20.988072 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:20.988077 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:20.989382 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:20.989383 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:20.990724 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:20.990727 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:20.991772 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:20.991773 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:20.991856 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:20.992214 (+ 358us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:10:20.992217 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:10:20.994090 (+ 1873us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:10:20.995943 (+ 1853us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:10:20.995943 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:10:20.995944 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:10:20.996135 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:10:20.996354 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:10:20.996355 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:10:20.996355 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:10:20.996410 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:10:20.996682 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:10:20.996683 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:10:20.996726 (+ 43us) client_negotiation.cc:769] Sending connection context
0926 01:10:20.996789 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 01:10:20.998686 (+ 1897us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"mutex_wait_us":1810,"thread_start_us":78,"threads_started":1}
I20260926 01:10:20.999018 6491 ts_manager.cc:194] Re-registered known tserver with Master: 2264518cb0fe444da3e1606269628249 (127.0.98.1:35643)
I20260926 01:10:20.999109 7084 ts_manager.cc:194] Registered new tserver with Master: 2264518cb0fe444da3e1606269628249 (127.0.98.1:35643)
I20260926 01:10:20.999404 6491 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:46683
I20260926 01:10:20.999878 7460 heartbeater.cc:347] Connected to a master server at 127.0.98.60:43103
I20260926 01:10:20.999931 7460 heartbeater.cc:464] Registering TS with master...
I20260926 01:10:21.000037 7460 heartbeater.cc:511] Master 127.0.98.60:43103 requested a full tablet report, sending...
I20260926 01:10:21.000399 7281 ts_manager.cc:194] Registered new tserver with Master: 2264518cb0fe444da3e1606269628249 (127.0.98.1:35643)
W20260926 01:10:21.073525 7468 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:10:21.073778 7468 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:10:21.073805 7468 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:10:21.075155 7468 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:10:21.075201 7468 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:10:21.075229 7468 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 01:10:21.076653 7468 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:39297
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=38159
--tserver_master_addrs=127.0.98.62:45453,127.0.98.61:42317,127.0.98.60:43103
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7468
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:10:21.076854 7468 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:10:21.077039 7468 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:10:21.081947 7473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:10:21.081967 7476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:10:21.082137 7468 server_base.cc:1034] running on GCE node
W20260926 01:10:21.081981 7474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:10:21.082358 7468 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:10:21.082590 7468 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:10:21.083765 7468 hybrid_clock.cc:648] HybridClock initialized: now 1790385021083737 us; error 39 us; skew 500 ppm
Sep 26 01:10:21 dist-test-slave-7b22 krb5kdc[6446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385021, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:10:21.086356 7468 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260926 01:10:21.086773 7468 webserver.cc:460] Webserver started at http://127.0.98.2:38159/ using document root <none> and password file <none>
I20260926 01:10:21.086992 7468 fs_manager.cc:359] Metadata directory not provided
I20260926 01:10:21.087050 7468 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:10:21.088492 7468 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:10:21.089210 7483 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:21.089452 7468 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:10:21.089538 7468 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/ts-1/wal
uuid: "1ffe4fc932934431b944d78dfe891007"
format_stamp: "Formatted at 2026-09-26 01:10:20 on dist-test-slave-7b22"
server_key: "5d16c6db766df4828323c7d06db0c4e8"
server_key_iv: "eb7998c144aed7bd52a253cf01920a32"
server_key_version: "encryptionkey@0"
I20260926 01:10:21.089974 7468 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:10:21.103557 7468 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:10:21.105746 7468 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:10:21.105950 7468 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:10:21.106190 7468 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:10:21.106511 7468 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:10:21.106544 7468 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:21.106567 7468 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:10:21.106583 7468 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:21.111958 7468 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39297
I20260926 01:10:21.112028 7596 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39297 every 8 connection(s)
I20260926 01:10:21.112332 7468 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/ts-1/data/info.pb
I20260926 01:10:21.116991 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 7468
I20260926 01:10:21.117153 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 6821
Sep 26 01:10:21 dist-test-slave-7b22 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385021, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 26 01:10:21 dist-test-slave-7b22 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385021, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 26 01:10:21 dist-test-slave-7b22 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385021, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 01:10:21.123529 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:43231
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:45453
--webserver_port=36287
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:45453,127.0.98.61:42317
--tserver_master_addrs=127.0.98.62:45453,127.0.98.61:42317,127.0.98.60:43103 with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 01:10:21.124883 7464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:21.113072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60085 (local address 127.0.98.60:43103)
0926 01:10:21.113123 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:21.113125 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:21.113326 (+ 201us) server_negotiation.cc:407] Connection header received
0926 01:10:21.114558 (+ 1232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:21.114560 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:21.114575 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:21.114623 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:21.115392 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:21.115883 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:21.117779 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:21.118133 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:21.118176 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:21.121281 (+ 3105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:10:21.121283 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:10:21.121283 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:10:21.121291 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:10:21.122866 (+ 1575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:21.123479 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:21.123480 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:21.123481 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:21.123543 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:21.124068 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:21.124070 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:21.124070 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:21.124235 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:10:21.124282 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:21.124692 (+ 410us) server_negotiation.cc:299] Negotiation successful
0926 01:10:21.124788 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 01:10:21.125051 7139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:21.113483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36865 (local address 127.0.98.61:42317)
0926 01:10:21.113538 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:21.113540 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:21.113549 (+ 9us) server_negotiation.cc:407] Connection header received
0926 01:10:21.114691 (+ 1142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:21.114693 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:21.114708 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:21.114757 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:21.115404 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:21.116144 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:21.118562 (+ 2418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:21.118841 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:21.118883 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:21.122236 (+ 3353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:10:21.122237 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:10:21.122238 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:10:21.122246 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:10:21.123971 (+ 1725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:21.124334 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:21.124335 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:21.124335 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:21.124391 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:21.124556 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:21.124556 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:21.124556 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:21.124673 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:10:21.124720 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:21.124906 (+ 186us) server_negotiation.cc:299] Negotiation successful
0926 01:10:21.124973 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 01:10:21.125607 7603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:21.113331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42317 (local address 127.0.98.2:36865)
0926 01:10:21.113446 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:21.113455 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:21.114341 (+ 886us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:21.114802 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:21.114803 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:21.115029 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:10:21.115281 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:21.115286 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:21.116221 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:21.116221 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:21.118481 (+ 2260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:21.118483 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:21.118961 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:21.118961 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:21.119022 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:21.119415 (+ 393us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:10:21.119420 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:10:21.122129 (+ 2709us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:10:21.124058 (+ 1929us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:10:21.124059 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:10:21.124060 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:10:21.124247 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:10:21.124449 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:10:21.124450 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:10:21.124450 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:10:21.124500 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:10:21.124754 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:10:21.124755 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:10:21.124800 (+ 45us) client_negotiation.cc:769] Sending connection context
0926 01:10:21.125175 (+ 375us) client_negotiation.cc:240] Negotiation successful
0926 01:10:21.125376 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"mutex_wait_us":5,"thread_start_us":61,"threads_started":1}
I20260926 01:10:21.125975 6544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:21.113043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43015 (local address 127.0.98.62:45453)
0926 01:10:21.113135 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:21.113137 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:21.113324 (+ 187us) server_negotiation.cc:407] Connection header received
0926 01:10:21.114441 (+ 1117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:21.114442 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:21.114455 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:21.114503 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:21.115365 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:21.116144 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:21.118441 (+ 2297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:21.118730 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:21.118802 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:21.123124 (+ 4322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:10:21.123126 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:10:21.123126 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:10:21.123135 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:10:21.124683 (+ 1548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:21.125060 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:21.125061 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:21.125062 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:21.125122 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:21.125311 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:21.125312 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:21.125312 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:21.125438 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:10:21.125495 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:21.125790 (+ 295us) server_negotiation.cc:299] Negotiation successful
0926 01:10:21.125892 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 01:10:21.127259 7601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:21.113053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45453 (local address 127.0.98.2:43015)
0926 01:10:21.113207 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:21.113248 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:21.114328 (+ 1080us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:21.114583 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:21.114587 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:21.115030 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:10:21.115280 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:21.115285 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:21.117123 (+ 1838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:21.117124 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:21.118354 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:21.118356 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:21.118858 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:21.118859 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:21.118929 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:21.119311 (+ 382us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:10:21.119316 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:10:21.123038 (+ 3722us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:10:21.124769 (+ 1731us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:10:21.124770 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:10:21.124770 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:10:21.124963 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:10:21.125192 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:10:21.125193 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:10:21.125194 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:10:21.125248 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:10:21.125519 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:10:21.125520 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:10:21.125559 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 01:10:21.125608 (+ 49us) client_negotiation.cc:240] Negotiation successful
0926 01:10:21.125702 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"mutex_wait_us":42,"thread_start_us":87,"threads_started":1}
I20260926 01:10:21.126067 7599 heartbeater.cc:347] Connected to a master server at 127.0.98.61:42317
I20260926 01:10:21.127434 7599 heartbeater.cc:464] Registering TS with master...
I20260926 01:10:21.127640 7599 heartbeater.cc:511] Master 127.0.98.61:42317 requested a full tablet report, sending...
I20260926 01:10:21.127998 7602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:21.113053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43103 (local address 127.0.98.2:60085)
0926 01:10:21.113222 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:21.113248 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:21.114321 (+ 1073us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:21.114636 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:21.114636 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:21.115029 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:10:21.115281 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:21.115286 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:21.115992 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:21.115992 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:21.117196 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:21.117198 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:21.118276 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:21.118277 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:21.118425 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:21.119001 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:10:21.119012 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:10:21.121157 (+ 2145us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:10:21.122942 (+ 1785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:10:21.122950 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:10:21.122953 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:10:21.123387 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:10:21.123843 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:10:21.123844 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:10:21.123844 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:10:21.123989 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:10:21.124331 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:10:21.124337 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:10:21.124511 (+ 174us) client_negotiation.cc:769] Sending connection context
0926 01:10:21.125444 (+ 933us) client_negotiation.cc:240] Negotiation successful
0926 01:10:21.125554 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"mutex_wait_us":48,"thread_start_us":100,"threads_started":1}
I20260926 01:10:21.128115 7084 ts_manager.cc:194] Registered new tserver with Master: 1ffe4fc932934431b944d78dfe891007 (127.0.98.2:39297)
I20260926 01:10:21.128332 7597 heartbeater.cc:347] Connected to a master server at 127.0.98.60:43103
I20260926 01:10:21.128360 7598 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45453
I20260926 01:10:21.128384 7597 heartbeater.cc:464] Registering TS with master...
I20260926 01:10:21.128500 7598 heartbeater.cc:464] Registering TS with master...
I20260926 01:10:21.128561 7597 heartbeater.cc:511] Master 127.0.98.60:43103 requested a full tablet report, sending...
I20260926 01:10:21.128572 7598 heartbeater.cc:511] Master 127.0.98.62:45453 requested a full tablet report, sending...
I20260926 01:10:21.128793 6491 ts_manager.cc:194] Re-registered known tserver with Master: 1ffe4fc932934431b944d78dfe891007 (127.0.98.2:39297)
I20260926 01:10:21.128875 7281 ts_manager.cc:194] Registered new tserver with Master: 1ffe4fc932934431b944d78dfe891007 (127.0.98.2:39297)
I20260926 01:10:21.129179 6491 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:43015
W20260926 01:10:21.206758 7604 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:10:21.206966 7604 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:10:21.206990 7604 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:10:21.208201 7604 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:10:21.208250 7604 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:10:21.208276 7604 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 01:10:21.209789 7604 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:43231
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=36287
--tserver_master_addrs=127.0.98.62:45453,127.0.98.61:42317,127.0.98.60:43103
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7604
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:10:21.210055 7604 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:10:21.210284 7604 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:10:21.215170 7612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:10:21.215175 7610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:10:21.215235 7609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:10:21.215276 7604 server_base.cc:1034] running on GCE node
I20260926 01:10:21.215562 7604 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:10:21.215777 7604 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:10:21.216915 7604 hybrid_clock.cc:648] HybridClock initialized: now 1790385021216899 us; error 32 us; skew 500 ppm
Sep 26 01:10:21 dist-test-slave-7b22 krb5kdc[6446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385021, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:10:21.219480 7604 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260926 01:10:21.219983 7604 webserver.cc:460] Webserver started at http://127.0.98.3:36287/ using document root <none> and password file <none>
I20260926 01:10:21.220186 7604 fs_manager.cc:359] Metadata directory not provided
I20260926 01:10:21.220247 7604 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:10:21.221560 7604 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:10:21.222162 7619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:21.222328 7604 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 01:10:21.222409 7604 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/ts-2/wal
uuid: "72a12d78947e445c85a8d6222cc136af"
format_stamp: "Formatted at 2026-09-26 01:10:20 on dist-test-slave-7b22"
server_key: "fb7f7d6f7d8015dea249e16f1437c8a1"
server_key_iv: "43588d170860f4bdd29d12ecea839a22"
server_key_version: "encryptionkey@0"
I20260926 01:10:21.222746 7604 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:10:21.238852 7604 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:10:21.241061 7604 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:10:21.241204 7604 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:10:21.241423 7604 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:10:21.241706 7604 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:10:21.241737 7604 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:21.241766 7604 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:10:21.241788 7604 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:21.247164 7604 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43231
I20260926 01:10:21.247215 7732 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43231 every 8 connection(s)
I20260926 01:10:21.247537 7604 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/ts-2/data/info.pb
I20260926 01:10:21.249533 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 7604
Sep 26 01:10:21 dist-test-slave-7b22 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385021, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 26 01:10:21 dist-test-slave-7b22 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385021, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 26 01:10:21 dist-test-slave-7b22 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385021, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260926 01:10:21.260933 7737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:21.248207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42317 (local address 127.0.98.3:45847)
0926 01:10:21.248365 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:21.248420 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:21.249430 (+ 1010us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:21.249736 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:21.249737 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:21.250694 (+ 957us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:10:21.250760 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:21.250763 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:21.251628 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:21.251629 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:21.252981 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:21.252984 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:21.253490 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:21.253491 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:21.253627 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:21.254246 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:10:21.254257 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:10:21.257651 (+ 3394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:10:21.259330 (+ 1679us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:10:21.259338 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:10:21.259341 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:10:21.259629 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:10:21.259851 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:10:21.259852 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:10:21.259852 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:10:21.259974 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:10:21.260228 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:10:21.260234 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:10:21.260403 (+ 169us) client_negotiation.cc:769] Sending connection context
0926 01:10:21.260507 (+ 104us) client_negotiation.cc:240] Negotiation successful
0926 01:10:21.260682 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"mutex_wait_us":208,"thread_start_us":86,"threads_started":1}
I20260926 01:10:21.261375 7139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:21.248196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45847 (local address 127.0.98.61:42317)
0926 01:10:21.248251 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:21.248254 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:21.248504 (+ 250us) server_negotiation.cc:407] Connection header received
0926 01:10:21.249532 (+ 1028us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:21.249534 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:21.249545 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:21.249594 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:21.250838 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:21.251551 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:21.253049 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:21.253356 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:21.253415 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:21.257734 (+ 4319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:10:21.257736 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:10:21.257736 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:10:21.257744 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:10:21.259256 (+ 1512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:21.259717 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:21.259718 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:21.259718 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:21.259785 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:21.260036 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:21.260037 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:21.260037 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:21.260155 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:10:21.260206 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:21.261197 (+ 991us) server_negotiation.cc:299] Negotiation successful
0926 01:10:21.261274 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 01:10:21.261420 7738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:21.248506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45453 (local address 127.0.98.3:49257)
0926 01:10:21.249696 (+ 1190us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:21.249705 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:21.249784 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:21.249987 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:21.249987 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:21.250247 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:10:21.250567 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:21.250573 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:21.251497 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:21.251498 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:21.253779 (+ 2281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:21.253781 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:21.254280 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:21.254281 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:21.254341 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:21.254696 (+ 355us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:10:21.254700 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:10:21.257335 (+ 2635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:10:21.259889 (+ 2554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:10:21.259890 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:10:21.259890 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:10:21.260307 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:10:21.260526 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:10:21.260526 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:10:21.260527 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:10:21.260807 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:10:21.261103 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:10:21.261103 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:10:21.261145 (+ 42us) client_negotiation.cc:769] Sending connection context
0926 01:10:21.261198 (+ 53us) client_negotiation.cc:240] Negotiation successful
0926 01:10:21.261290 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1073,"mutex_wait_us":1305,"thread_start_us":52,"threads_started":1}
I20260926 01:10:21.261695 7735 heartbeater.cc:347] Connected to a master server at 127.0.98.61:42317
I20260926 01:10:21.261749 6544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:21.248578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49257 (local address 127.0.98.62:45453)
0926 01:10:21.248631 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:21.248633 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:21.249750 (+ 1117us) server_negotiation.cc:407] Connection header received
0926 01:10:21.249883 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:21.249885 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:21.249897 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:21.249926 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:21.250896 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:21.251389 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:21.253891 (+ 2502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:21.254172 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:21.254229 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:21.257430 (+ 3201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:10:21.257430 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:10:21.257431 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:10:21.257438 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:10:21.259801 (+ 2363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:21.260397 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:21.260398 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:21.260399 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:21.260459 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:21.260880 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:21.260881 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:21.260881 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:21.261016 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:10:21.261078 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:21.261587 (+ 509us) server_negotiation.cc:299] Negotiation successful
0926 01:10:21.261665 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 01:10:21.261790 7735 heartbeater.cc:464] Registering TS with master...
I20260926 01:10:21.262010 7735 heartbeater.cc:511] Master 127.0.98.61:42317 requested a full tablet report, sending...
I20260926 01:10:21.262485 7734 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45453
I20260926 01:10:21.262539 7734 heartbeater.cc:464] Registering TS with master...
I20260926 01:10:21.262601 7734 heartbeater.cc:511] Master 127.0.98.62:45453 requested a full tablet report, sending...
I20260926 01:10:21.262610 7464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:21.248267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37763 (local address 127.0.98.60:43103)
0926 01:10:21.248311 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:21.248312 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:21.248661 (+ 349us) server_negotiation.cc:407] Connection header received
0926 01:10:21.249547 (+ 886us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:21.249549 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:21.249560 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:21.249589 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:21.250702 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:21.251508 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:21.253076 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:21.253375 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:21.253415 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:21.258307 (+ 4892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:10:21.258308 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:10:21.258308 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:10:21.258315 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:10:21.259801 (+ 1486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:21.261800 (+ 1999us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:21.261801 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:21.261802 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:21.261887 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:21.262080 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:21.262081 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:21.262081 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:21.262212 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:10:21.262254 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:21.262439 (+ 185us) server_negotiation.cc:299] Negotiation successful
0926 01:10:21.262527 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 01:10:21.262863 7084 ts_manager.cc:194] Registered new tserver with Master: 72a12d78947e445c85a8d6222cc136af (127.0.98.3:43231)
I20260926 01:10:21.262907 7739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:21.248207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43103 (local address 127.0.98.3:37763)
0926 01:10:21.248608 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:21.248617 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:21.249448 (+ 831us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:21.249731 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:21.249736 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:21.250249 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:10:21.250567 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:21.250573 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:21.251641 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:21.251641 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:21.252983 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:21.252986 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:21.253530 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:21.253530 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:21.253627 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:21.254957 (+ 1330us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:10:21.254962 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:10:21.258164 (+ 3202us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:10:21.261514 (+ 3350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:10:21.261515 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:10:21.261515 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:10:21.261704 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:10:21.261959 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:10:21.261959 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:10:21.261960 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:10:21.262013 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:10:21.262296 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:10:21.262297 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:10:21.262338 (+ 41us) client_negotiation.cc:769] Sending connection context
0926 01:10:21.262703 (+ 365us) client_negotiation.cc:240] Negotiation successful
0926 01:10:21.262814 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":374,"mutex_wait_us":29,"thread_start_us":89,"threads_started":1}
I20260926 01:10:21.263221 7733 heartbeater.cc:347] Connected to a master server at 127.0.98.60:43103
I20260926 01:10:21.263343 7733 heartbeater.cc:464] Registering TS with master...
I20260926 01:10:21.263441 7733 heartbeater.cc:511] Master 127.0.98.60:43103 requested a full tablet report, sending...
I20260926 01:10:21.263710 7746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:21.250722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49110 (local address 127.0.98.60:43103)
0926 01:10:21.250866 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:21.250868 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:21.251986 (+ 1118us) server_negotiation.cc:407] Connection header received
0926 01:10:21.252081 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:21.252083 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:21.252094 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:21.252179 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:21.253884 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:21.254569 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:21.255670 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:21.255965 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:21.256023 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:21.258048 (+ 2025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:10:21.258049 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:10:21.258050 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:10:21.258056 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:10:21.260846 (+ 2790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:21.262120 (+ 1274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:21.262121 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:21.262122 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:21.262180 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:21.262371 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:21.262371 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:21.262372 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:21.262496 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:10:21.263495 (+ 999us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:21.263533 (+ 38us) server_negotiation.cc:299] Negotiation successful
0926 01:10:21.263611 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":42,"threads_started":1}
I20260926 01:10:21.263823 7281 ts_manager.cc:194] Registered new tserver with Master: 72a12d78947e445c85a8d6222cc136af (127.0.98.3:43231)
I20260926 01:10:21.263826 6491 ts_manager.cc:194] Re-registered known tserver with Master: 72a12d78947e445c85a8d6222cc136af (127.0.98.3:43231)
I20260926 01:10:21.264329 6491 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:49257
I20260926 01:10:21.264462 7748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:21.251558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39600 (local address 127.0.98.62:45453)
0926 01:10:21.251819 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:21.251820 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:21.251919 (+ 99us) server_negotiation.cc:407] Connection header received
0926 01:10:21.251943 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:21.251943 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:21.251955 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:21.252556 (+ 601us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:21.253294 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:21.254438 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:21.255722 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:21.256012 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:21.256061 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:21.257250 (+ 1189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:10:21.257251 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:10:21.257251 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:10:21.257259 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:10:21.263142 (+ 5883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:21.263472 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:21.263473 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:21.263473 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:21.263529 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:21.263742 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:21.263743 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:21.263743 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:21.263916 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:10:21.264009 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:21.264306 (+ 297us) server_negotiation.cc:299] Negotiation successful
0926 01:10:21.264384 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":90,"threads_started":1}
I20260926 01:10:21.266031 6491 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39600:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:10:21.267621 6491 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:10:21.267756 7749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:21.251876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32842 (local address 127.0.98.61:42317)
0926 01:10:21.254504 (+ 2628us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:21.254505 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:21.254512 (+ 7us) server_negotiation.cc:407] Connection header received
0926 01:10:21.254530 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:21.254531 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:21.254541 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:21.258234 (+ 3693us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:21.258369 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:21.259078 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:21.260026 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:21.260320 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:21.260371 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:21.265369 (+ 4998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:10:21.265370 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:10:21.265371 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:10:21.265378 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:10:21.266627 (+ 1249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:21.266940 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:21.266941 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:21.266941 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:21.267000 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:21.267158 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:21.267159 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:21.267159 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:21.267277 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:10:21.267362 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:21.267579 (+ 217us) server_negotiation.cc:299] Negotiation successful
0926 01:10:21.267663 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2608,"thread_start_us":68,"threads_started":1}
I20260926 01:10:21.274776 7754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:21.271262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43231 (local address 127.0.0.1:53170)
0926 01:10:21.271630 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:21.271638 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:21.271670 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:21.271924 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:21.271925 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:21.271928 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:10:21.271971 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:21.271973 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:21.272786 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:21.272787 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:21.273591 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:21.273594 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:21.274517 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:21.274518 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:21.274580 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:21.274586 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 01:10:21.274640 (+ 54us) client_negotiation.cc:240] Negotiation successful
0926 01:10:21.274690 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356,"mutex_wait_us":119,"thread_start_us":78,"threads_started":1}
I20260926 01:10:21.274781 7757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:21.271439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53170 (local address 127.0.98.3:43231)
0926 01:10:21.271734 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:21.271738 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:21.271748 (+ 10us) server_negotiation.cc:407] Connection header received
0926 01:10:21.271782 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:21.271784 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:21.271799 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:21.271924 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:10:21.272022 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:21.272709 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:21.273677 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:21.274444 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:21.274510 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:21.274590 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:21.274656 (+ 66us) server_negotiation.cc:299] Negotiation successful
0926 01:10:21.274698 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":82,"threads_started":1}
I20260926 01:10:21.275501 7667 tablet_service.cc:1467] Processing CreateTablet for tablet e12eb2c62b114733ac41963b3d8cb886 (DEFAULT_TABLE table=test-table [id=e449df1c119d4d4ea0f0ac1c0b5ff7af]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:10:21.275893 7048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:21.271262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39297 (local address 127.0.0.1:41022)
0926 01:10:21.271297 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:21.271307 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:21.271341 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:21.272005 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:21.272006 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:21.272007 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:10:21.272035 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:21.272036 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:21.273255 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:21.273256 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:21.274541 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:21.274544 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:21.275678 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:21.275680 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:21.275738 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:21.275741 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 01:10:21.275780 (+ 39us) client_negotiation.cc:240] Negotiation successful
0926 01:10:21.275824 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19,"mutex_wait_us":90}
I20260926 01:10:21.275894 7667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e12eb2c62b114733ac41963b3d8cb886. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:10:21.276081 7755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:21.271303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35643 (local address 127.0.0.1:60344)
0926 01:10:21.271510 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:21.271519 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:21.271561 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:21.271883 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:21.271884 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:21.271885 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:10:21.271940 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:21.271941 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:21.273254 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:21.273255 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:21.274546 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:21.274547 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:21.275561 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:21.275562 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:21.275621 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:21.275626 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 01:10:21.275685 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 01:10:21.275724 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"mutex_wait_us":46,"thread_start_us":85,"threads_started":1}
I20260926 01:10:21.276183 7758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:21.271515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60344 (local address 127.0.98.1:35643)
0926 01:10:21.271668 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:21.271673 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:21.271683 (+ 10us) server_negotiation.cc:407] Connection header received
0926 01:10:21.271721 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:21.271723 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:21.271739 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:21.271811 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:10:21.272008 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:21.273133 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:21.274636 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:21.275480 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:21.275551 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:21.275626 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:21.276062 (+ 436us) server_negotiation.cc:299] Negotiation successful
0926 01:10:21.276105 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":76,"threads_started":1}
I20260926 01:10:21.276649 7756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:21.271432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41022 (local address 127.0.98.2:39297)
0926 01:10:21.271851 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:21.271855 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:21.271863 (+ 8us) server_negotiation.cc:407] Connection header received
0926 01:10:21.271897 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:21.271899 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:21.271913 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:21.272095 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:10:21.272147 (+ 52us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:21.273174 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:21.274634 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:21.275584 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:21.275747 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:21.276529 (+ 782us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:21.276553 (+ 24us) server_negotiation.cc:299] Negotiation successful
0926 01:10:21.276590 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":75,"threads_started":1}
I20260926 01:10:21.276830 7394 tablet_service.cc:1467] Processing CreateTablet for tablet e12eb2c62b114733ac41963b3d8cb886 (DEFAULT_TABLE table=test-table [id=e449df1c119d4d4ea0f0ac1c0b5ff7af]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:10:21.277163 7394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e12eb2c62b114733ac41963b3d8cb886. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:10:21.277228 7531 tablet_service.cc:1467] Processing CreateTablet for tablet e12eb2c62b114733ac41963b3d8cb886 (DEFAULT_TABLE table=test-table [id=e449df1c119d4d4ea0f0ac1c0b5ff7af]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:10:21.277515 7531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e12eb2c62b114733ac41963b3d8cb886. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:10:21.279476 7759 tablet_bootstrap.cc:492] T e12eb2c62b114733ac41963b3d8cb886 P 72a12d78947e445c85a8d6222cc136af: Bootstrap starting.
I20260926 01:10:21.280138 7759 tablet_bootstrap.cc:654] T e12eb2c62b114733ac41963b3d8cb886 P 72a12d78947e445c85a8d6222cc136af: Neither blocks nor log segments found. Creating new log.
I20260926 01:10:21.280447 7759 log.cc:824] T e12eb2c62b114733ac41963b3d8cb886 P 72a12d78947e445c85a8d6222cc136af: Log is configured to *not* fsync() on all Append() calls
I20260926 01:10:21.280781 7760 tablet_bootstrap.cc:492] T e12eb2c62b114733ac41963b3d8cb886 P 1ffe4fc932934431b944d78dfe891007: Bootstrap starting.
I20260926 01:10:21.281129 7762 tablet_bootstrap.cc:492] T e12eb2c62b114733ac41963b3d8cb886 P 2264518cb0fe444da3e1606269628249: Bootstrap starting.
I20260926 01:10:21.281224 7759 tablet_bootstrap.cc:492] T e12eb2c62b114733ac41963b3d8cb886 P 72a12d78947e445c85a8d6222cc136af: No bootstrap required, opened a new log
I20260926 01:10:21.281277 7759 ts_tablet_manager.cc:1397] T e12eb2c62b114733ac41963b3d8cb886 P 72a12d78947e445c85a8d6222cc136af: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 01:10:21.281713 7762 tablet_bootstrap.cc:654] T e12eb2c62b114733ac41963b3d8cb886 P 2264518cb0fe444da3e1606269628249: Neither blocks nor log segments found. Creating new log.
I20260926 01:10:21.282032 7762 log.cc:824] T e12eb2c62b114733ac41963b3d8cb886 P 2264518cb0fe444da3e1606269628249: Log is configured to *not* fsync() on all Append() calls
I20260926 01:10:21.282363 7760 tablet_bootstrap.cc:654] T e12eb2c62b114733ac41963b3d8cb886 P 1ffe4fc932934431b944d78dfe891007: Neither blocks nor log segments found. Creating new log.
I20260926 01:10:21.282617 7760 log.cc:824] T e12eb2c62b114733ac41963b3d8cb886 P 1ffe4fc932934431b944d78dfe891007: Log is configured to *not* fsync() on all Append() calls
I20260926 01:10:21.283265 7760 tablet_bootstrap.cc:492] T e12eb2c62b114733ac41963b3d8cb886 P 1ffe4fc932934431b944d78dfe891007: No bootstrap required, opened a new log
I20260926 01:10:21.283318 7760 ts_tablet_manager.cc:1397] T e12eb2c62b114733ac41963b3d8cb886 P 1ffe4fc932934431b944d78dfe891007: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260926 01:10:21.283532 7759 raft_consensus.cc:348] T e12eb2c62b114733ac41963b3d8cb886 P 72a12d78947e445c85a8d6222cc136af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72a12d78947e445c85a8d6222cc136af" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43231 } } peers { permanent_uuid: "1ffe4fc932934431b944d78dfe891007" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39297 } } peers { permanent_uuid: "2264518cb0fe444da3e1606269628249" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35643 } }
I20260926 01:10:21.283664 7759 raft_consensus.cc:374] T e12eb2c62b114733ac41963b3d8cb886 P 72a12d78947e445c85a8d6222cc136af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:10:21.283715 7759 raft_consensus.cc:729] T e12eb2c62b114733ac41963b3d8cb886 P 72a12d78947e445c85a8d6222cc136af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72a12d78947e445c85a8d6222cc136af, State: Initialized, Role: FOLLOWER
I20260926 01:10:21.283831 7759 consensus_queue.cc:260] T e12eb2c62b114733ac41963b3d8cb886 P 72a12d78947e445c85a8d6222cc136af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72a12d78947e445c85a8d6222cc136af" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43231 } } peers { permanent_uuid: "1ffe4fc932934431b944d78dfe891007" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39297 } } peers { permanent_uuid: "2264518cb0fe444da3e1606269628249" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35643 } }
I20260926 01:10:21.284269 7734 heartbeater.cc:503] Master 127.0.98.62:45453 was elected leader, sending a full tablet report...
I20260926 01:10:21.284899 7759 ts_tablet_manager.cc:1428] T e12eb2c62b114733ac41963b3d8cb886 P 72a12d78947e445c85a8d6222cc136af: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260926 01:10:21.284942 7760 raft_consensus.cc:348] T e12eb2c62b114733ac41963b3d8cb886 P 1ffe4fc932934431b944d78dfe891007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72a12d78947e445c85a8d6222cc136af" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43231 } } peers { permanent_uuid: "1ffe4fc932934431b944d78dfe891007" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39297 } } peers { permanent_uuid: "2264518cb0fe444da3e1606269628249" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35643 } }
I20260926 01:10:21.285085 7760 raft_consensus.cc:374] T e12eb2c62b114733ac41963b3d8cb886 P 1ffe4fc932934431b944d78dfe891007 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:10:21.285120 7760 raft_consensus.cc:729] T e12eb2c62b114733ac41963b3d8cb886 P 1ffe4fc932934431b944d78dfe891007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ffe4fc932934431b944d78dfe891007, State: Initialized, Role: FOLLOWER
I20260926 01:10:21.285250 7760 consensus_queue.cc:260] T e12eb2c62b114733ac41963b3d8cb886 P 1ffe4fc932934431b944d78dfe891007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72a12d78947e445c85a8d6222cc136af" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43231 } } peers { permanent_uuid: "1ffe4fc932934431b944d78dfe891007" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39297 } } peers { permanent_uuid: "2264518cb0fe444da3e1606269628249" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35643 } }
I20260926 01:10:21.285498 7760 ts_tablet_manager.cc:1428] T e12eb2c62b114733ac41963b3d8cb886 P 1ffe4fc932934431b944d78dfe891007: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 01:10:21.285712 7598 heartbeater.cc:503] Master 127.0.98.62:45453 was elected leader, sending a full tablet report...
I20260926 01:10:21.286234 7762 tablet_bootstrap.cc:492] T e12eb2c62b114733ac41963b3d8cb886 P 2264518cb0fe444da3e1606269628249: No bootstrap required, opened a new log
I20260926 01:10:21.286306 7762 ts_tablet_manager.cc:1397] T e12eb2c62b114733ac41963b3d8cb886 P 2264518cb0fe444da3e1606269628249: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20260926 01:10:21.288249 7762 raft_consensus.cc:348] T e12eb2c62b114733ac41963b3d8cb886 P 2264518cb0fe444da3e1606269628249 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72a12d78947e445c85a8d6222cc136af" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43231 } } peers { permanent_uuid: "1ffe4fc932934431b944d78dfe891007" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39297 } } peers { permanent_uuid: "2264518cb0fe444da3e1606269628249" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35643 } }
I20260926 01:10:21.288376 7762 raft_consensus.cc:374] T e12eb2c62b114733ac41963b3d8cb886 P 2264518cb0fe444da3e1606269628249 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:10:21.288396 7762 raft_consensus.cc:729] T e12eb2c62b114733ac41963b3d8cb886 P 2264518cb0fe444da3e1606269628249 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2264518cb0fe444da3e1606269628249, State: Initialized, Role: FOLLOWER
I20260926 01:10:21.288502 7762 consensus_queue.cc:260] T e12eb2c62b114733ac41963b3d8cb886 P 2264518cb0fe444da3e1606269628249 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72a12d78947e445c85a8d6222cc136af" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43231 } } peers { permanent_uuid: "1ffe4fc932934431b944d78dfe891007" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39297 } } peers { permanent_uuid: "2264518cb0fe444da3e1606269628249" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35643 } }
I20260926 01:10:21.288724 7762 ts_tablet_manager.cc:1428] T e12eb2c62b114733ac41963b3d8cb886 P 2264518cb0fe444da3e1606269628249: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260926 01:10:21.288854 7461 heartbeater.cc:503] Master 127.0.98.62:45453 was elected leader, sending a full tablet report...
W20260926 01:10:21.363315 7600 tablet.cc:2369] T e12eb2c62b114733ac41963b3d8cb886 P 1ffe4fc932934431b944d78dfe891007: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 01:10:21.455471 7746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:21.452914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49116 (local address 127.0.98.60:43103)
0926 01:10:21.452971 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:21.452972 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:21.452983 (+ 11us) server_negotiation.cc:407] Connection header received
0926 01:10:21.453011 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:21.453013 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:21.453031 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:21.453104 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:10:21.453239 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:21.453742 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:21.454780 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:21.455193 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:21.455247 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:21.455302 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:21.455381 (+ 79us) server_negotiation.cc:299] Negotiation successful
0926 01:10:21.455412 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 01:10:21.455479 7755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:21.452806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43103 (local address 127.0.0.1:49116)
0926 01:10:21.452885 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:21.452897 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:21.452949 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:21.453114 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:21.453114 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:21.453116 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:10:21.453173 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:21.453176 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:21.453858 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:21.453859 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:21.454686 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:21.454689 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:21.455270 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:21.455271 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:21.455322 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:21.455325 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 01:10:21.455373 (+ 48us) client_negotiation.cc:240] Negotiation successful
0926 01:10:21.455415 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260926 01:10:21.455817 7291 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 3b48fee3b6474a23a70b62be3d2b7f2f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.9] Dedup: 1.7->[1.8-1.9]
I20260926 01:10:21.456199 7768 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b48fee3b6474a23a70b62be3d2b7f2f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5afbc257ec7449b189377b1ae0cbebd5. Latest consensus state: current_term: 1 leader_uuid: "5afbc257ec7449b189377b1ae0cbebd5" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5afbc257ec7449b189377b1ae0cbebd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45453 } } peers { permanent_uuid: "8971f3cf560c49ae9dd58727c3b669c4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42317 } attrs { promote: false } } peers { permanent_uuid: "3b48fee3b6474a23a70b62be3d2b7f2f" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43103 } attrs { promote: true } } }
I20260926 01:10:21.456324 7768 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b48fee3b6474a23a70b62be3d2b7f2f [sys.catalog]: This master's current role is: LEARNER
W20260926 01:10:21.483762 7463 tablet.cc:2369] T e12eb2c62b114733ac41963b3d8cb886 P 2264518cb0fe444da3e1606269628249: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 01:10:21.498306 7736 tablet.cc:2369] T e12eb2c62b114733ac41963b3d8cb886 P 72a12d78947e445c85a8d6222cc136af: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 01:10:21.533041 7765 raft_consensus.cc:482] T e12eb2c62b114733ac41963b3d8cb886 P 72a12d78947e445c85a8d6222cc136af [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:10:21.533165 7765 raft_consensus.cc:504] T e12eb2c62b114733ac41963b3d8cb886 P 72a12d78947e445c85a8d6222cc136af [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72a12d78947e445c85a8d6222cc136af" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43231 } } peers { permanent_uuid: "1ffe4fc932934431b944d78dfe891007" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39297 } } peers { permanent_uuid: "2264518cb0fe444da3e1606269628249" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35643 } }
I20260926 01:10:21.533484 7765 leader_election.cc:288] T e12eb2c62b114733ac41963b3d8cb886 P 72a12d78947e445c85a8d6222cc136af [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1ffe4fc932934431b944d78dfe891007 (127.0.98.2:39297), 2264518cb0fe444da3e1606269628249 (127.0.98.1:35643)
I20260926 01:10:21.536131 7758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:21.533753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42917 (local address 127.0.98.1:35643)
0926 01:10:21.533794 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:21.533796 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:21.533805 (+ 9us) server_negotiation.cc:407] Connection header received
0926 01:10:21.533905 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:21.533906 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:21.533918 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:21.533963 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:10:21.534099 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:21.534563 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:21.535489 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:21.535838 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:21.535895 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:21.535905 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:21.536027 (+ 122us) server_negotiation.cc:299] Negotiation successful
0926 01:10:21.536067 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 01:10:21.536216 7738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:21.533673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39297 (local address 127.0.98.3:48161)
0926 01:10:21.533845 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:21.533854 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:21.533904 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:21.534027 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:21.534028 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:21.534029 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:10:21.534102 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:21.534103 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:21.534747 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:21.534748 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:21.535430 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:21.535432 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:21.535935 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:21.535936 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:21.536003 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:21.536006 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 01:10:21.536048 (+ 42us) client_negotiation.cc:240] Negotiation successful
0926 01:10:21.536087 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"mutex_wait_us":110}
I20260926 01:10:21.536314 7756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:21.533728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48161 (local address 127.0.98.2:39297)
0926 01:10:21.533789 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:21.533791 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:21.533892 (+ 101us) server_negotiation.cc:407] Connection header received
0926 01:10:21.533955 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:21.533956 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:21.533967 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:21.534015 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:10:21.534158 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:21.534613 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:21.535515 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:21.535861 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:21.535919 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:21.535929 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:21.536214 (+ 285us) server_negotiation.cc:299] Negotiation successful
0926 01:10:21.536255 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 01:10:21.536363 7739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:21.533665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35643 (local address 127.0.98.3:42917)
0926 01:10:21.533744 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:21.533756 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:21.533802 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:21.533972 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:21.533973 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:21.533975 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:10:21.534040 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:21.534042 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:21.534634 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:21.534635 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:21.535403 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:21.535405 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:21.535903 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:21.535904 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:21.535969 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:21.535972 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 01:10:21.536015 (+ 43us) client_negotiation.cc:240] Negotiation successful
0926 01:10:21.536056 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260926 01:10:21.536585 7551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e12eb2c62b114733ac41963b3d8cb886" candidate_uuid: "72a12d78947e445c85a8d6222cc136af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ffe4fc932934431b944d78dfe891007" is_pre_election: true
I20260926 01:10:21.536741 7551 raft_consensus.cc:2454] T e12eb2c62b114733ac41963b3d8cb886 P 1ffe4fc932934431b944d78dfe891007 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 72a12d78947e445c85a8d6222cc136af in term 0.
I20260926 01:10:21.536957 7622 leader_election.cc:302] T e12eb2c62b114733ac41963b3d8cb886 P 72a12d78947e445c85a8d6222cc136af [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ffe4fc932934431b944d78dfe891007, 72a12d78947e445c85a8d6222cc136af; no voters:
I20260926 01:10:21.537082 7765 raft_consensus.cc:2781] T e12eb2c62b114733ac41963b3d8cb886 P 72a12d78947e445c85a8d6222cc136af [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:10:21.537154 7765 raft_consensus.cc:482] T e12eb2c62b114733ac41963b3d8cb886 P 72a12d78947e445c85a8d6222cc136af [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:10:21.537108 7414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e12eb2c62b114733ac41963b3d8cb886" candidate_uuid: "72a12d78947e445c85a8d6222cc136af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2264518cb0fe444da3e1606269628249" is_pre_election: true
I20260926 01:10:21.537184 7765 raft_consensus.cc:3037] T e12eb2c62b114733ac41963b3d8cb886 P 72a12d78947e445c85a8d6222cc136af [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:10:21.537246 7414 raft_consensus.cc:2454] T e12eb2c62b114733ac41963b3d8cb886 P 2264518cb0fe444da3e1606269628249 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 72a12d78947e445c85a8d6222cc136af in term 0.
I20260926 01:10:21.537920 7765 raft_consensus.cc:504] T e12eb2c62b114733ac41963b3d8cb886 P 72a12d78947e445c85a8d6222cc136af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72a12d78947e445c85a8d6222cc136af" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43231 } } peers { permanent_uuid: "1ffe4fc932934431b944d78dfe891007" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39297 } } peers { permanent_uuid: "2264518cb0fe444da3e1606269628249" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35643 } }
I20260926 01:10:21.538038 7765 leader_election.cc:288] T e12eb2c62b114733ac41963b3d8cb886 P 72a12d78947e445c85a8d6222cc136af [CANDIDATE]: Term 1 election: Requested vote from peers 1ffe4fc932934431b944d78dfe891007 (127.0.98.2:39297), 2264518cb0fe444da3e1606269628249 (127.0.98.1:35643)
I20260926 01:10:21.538183 7414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e12eb2c62b114733ac41963b3d8cb886" candidate_uuid: "72a12d78947e445c85a8d6222cc136af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2264518cb0fe444da3e1606269628249"
I20260926 01:10:21.538178 7551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e12eb2c62b114733ac41963b3d8cb886" candidate_uuid: "72a12d78947e445c85a8d6222cc136af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ffe4fc932934431b944d78dfe891007"
I20260926 01:10:21.538251 7414 raft_consensus.cc:3037] T e12eb2c62b114733ac41963b3d8cb886 P 2264518cb0fe444da3e1606269628249 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:10:21.538256 7551 raft_consensus.cc:3037] T e12eb2c62b114733ac41963b3d8cb886 P 1ffe4fc932934431b944d78dfe891007 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:10:21.538872 7551 raft_consensus.cc:2454] T e12eb2c62b114733ac41963b3d8cb886 P 1ffe4fc932934431b944d78dfe891007 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72a12d78947e445c85a8d6222cc136af in term 1.
I20260926 01:10:21.538916 7414 raft_consensus.cc:2454] T e12eb2c62b114733ac41963b3d8cb886 P 2264518cb0fe444da3e1606269628249 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72a12d78947e445c85a8d6222cc136af in term 1.
I20260926 01:10:21.539027 7622 leader_election.cc:302] T e12eb2c62b114733ac41963b3d8cb886 P 72a12d78947e445c85a8d6222cc136af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ffe4fc932934431b944d78dfe891007, 72a12d78947e445c85a8d6222cc136af; no voters:
I20260926 01:10:21.539113 7765 raft_consensus.cc:2781] T e12eb2c62b114733ac41963b3d8cb886 P 72a12d78947e445c85a8d6222cc136af [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:10:21.539251 7765 raft_consensus.cc:686] T e12eb2c62b114733ac41963b3d8cb886 P 72a12d78947e445c85a8d6222cc136af [term 1 LEADER]: Becoming Leader. State: Replica: 72a12d78947e445c85a8d6222cc136af, State: Running, Role: LEADER
I20260926 01:10:21.539332 7765 consensus_queue.cc:237] T e12eb2c62b114733ac41963b3d8cb886 P 72a12d78947e445c85a8d6222cc136af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72a12d78947e445c85a8d6222cc136af" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43231 } } peers { permanent_uuid: "1ffe4fc932934431b944d78dfe891007" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39297 } } peers { permanent_uuid: "2264518cb0fe444da3e1606269628249" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35643 } }
I20260926 01:10:21.540082 6491 catalog_manager.cc:5495] T e12eb2c62b114733ac41963b3d8cb886 P 72a12d78947e445c85a8d6222cc136af reported cstate change: term changed from 0 to 1, leader changed from <none> to 72a12d78947e445c85a8d6222cc136af (127.0.98.3). New cstate: current_term: 1 leader_uuid: "72a12d78947e445c85a8d6222cc136af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72a12d78947e445c85a8d6222cc136af" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ffe4fc932934431b944d78dfe891007" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39297 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2264518cb0fe444da3e1606269628249" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35643 } health_report { overall_health: UNKNOWN } } }
I20260926 01:10:21.540589 7050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5afbc257ec7449b189377b1ae0cbebd5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5afbc257ec7449b189377b1ae0cbebd5" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5afbc257ec7449b189377b1ae0cbebd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45453 } } peers { permanent_uuid: "8971f3cf560c49ae9dd58727c3b669c4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42317 } attrs { promote: false } } peers { permanent_uuid: "3b48fee3b6474a23a70b62be3d2b7f2f" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43103 } attrs { promote: true } } }
I20260926 01:10:21.540675 7050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5afbc257ec7449b189377b1ae0cbebd5 [sys.catalog]: This master's current role is: LEADER
I20260926 01:10:21.541368 7049 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 5afbc257ec7449b189377b1ae0cbebd5: attempting to promote NON_VOTER 3b48fee3b6474a23a70b62be3d2b7f2f to VOTER
I20260926 01:10:21.541474 7049 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5afbc257ec7449b189377b1ae0cbebd5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "5afbc257ec7449b189377b1ae0cbebd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45453 } } peers { permanent_uuid: "8971f3cf560c49ae9dd58727c3b669c4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42317 } attrs { promote: false } } peers { permanent_uuid: "3b48fee3b6474a23a70b62be3d2b7f2f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43103 } attrs { promote: false } }
I20260926 01:10:21.541878 7098 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8971f3cf560c49ae9dd58727c3b669c4 [term 1 FOLLOWER]: Refusing update from remote peer 5afbc257ec7449b189377b1ae0cbebd5: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260926 01:10:21.541898 7291 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3b48fee3b6474a23a70b62be3d2b7f2f [term 1 LEARNER]: Refusing update from remote peer 5afbc257ec7449b189377b1ae0cbebd5: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260926 01:10:21.542085 7050 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5afbc257ec7449b189377b1ae0cbebd5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8971f3cf560c49ae9dd58727c3b669c4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42317 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260926 01:10:21.542192 7242 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5afbc257ec7449b189377b1ae0cbebd5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b48fee3b6474a23a70b62be3d2b7f2f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43103 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260926 01:10:21.542621 7050 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5afbc257ec7449b189377b1ae0cbebd5 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 3b48fee3b6474a23a70b62be3d2b7f2f (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "5afbc257ec7449b189377b1ae0cbebd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45453 } } peers { permanent_uuid: "8971f3cf560c49ae9dd58727c3b669c4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42317 } attrs { promote: false } } peers { permanent_uuid: "3b48fee3b6474a23a70b62be3d2b7f2f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43103 } attrs { promote: false } } }
I20260926 01:10:21.542760 7098 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8971f3cf560c49ae9dd58727c3b669c4 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 3b48fee3b6474a23a70b62be3d2b7f2f (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "5afbc257ec7449b189377b1ae0cbebd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45453 } } peers { permanent_uuid: "8971f3cf560c49ae9dd58727c3b669c4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42317 } attrs { promote: false } } peers { permanent_uuid: "3b48fee3b6474a23a70b62be3d2b7f2f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43103 } attrs { promote: false } } }
I20260926 01:10:21.542908 7291 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3b48fee3b6474a23a70b62be3d2b7f2f [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 3b48fee3b6474a23a70b62be3d2b7f2f (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "5afbc257ec7449b189377b1ae0cbebd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45453 } } peers { permanent_uuid: "8971f3cf560c49ae9dd58727c3b669c4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42317 } attrs { promote: false } } peers { permanent_uuid: "3b48fee3b6474a23a70b62be3d2b7f2f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43103 } attrs { promote: false } } }
I20260926 01:10:21.543517 7050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5afbc257ec7449b189377b1ae0cbebd5 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "5afbc257ec7449b189377b1ae0cbebd5" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "5afbc257ec7449b189377b1ae0cbebd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45453 } } peers { permanent_uuid: "8971f3cf560c49ae9dd58727c3b669c4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42317 } attrs { promote: false } } peers { permanent_uuid: "3b48fee3b6474a23a70b62be3d2b7f2f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43103 } attrs { promote: false } } }
I20260926 01:10:21.543591 7050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5afbc257ec7449b189377b1ae0cbebd5 [sys.catalog]: This master's current role is: LEADER
I20260926 01:10:21.543638 7050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5afbc257ec7449b189377b1ae0cbebd5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5afbc257ec7449b189377b1ae0cbebd5" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "5afbc257ec7449b189377b1ae0cbebd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45453 } } peers { permanent_uuid: "8971f3cf560c49ae9dd58727c3b669c4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42317 } attrs { promote: false } } peers { permanent_uuid: "3b48fee3b6474a23a70b62be3d2b7f2f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43103 } attrs { promote: false } } }
I20260926 01:10:21.543682 7050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5afbc257ec7449b189377b1ae0cbebd5 [sys.catalog]: This master's current role is: LEADER
I20260926 01:10:21.543730 7050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5afbc257ec7449b189377b1ae0cbebd5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5afbc257ec7449b189377b1ae0cbebd5" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "5afbc257ec7449b189377b1ae0cbebd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45453 } } peers { permanent_uuid: "8971f3cf560c49ae9dd58727c3b669c4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42317 } attrs { promote: false } } peers { permanent_uuid: "3b48fee3b6474a23a70b62be3d2b7f2f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43103 } attrs { promote: false } } }
I20260926 01:10:21.543773 7050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5afbc257ec7449b189377b1ae0cbebd5 [sys.catalog]: This master's current role is: LEADER
I20260926 01:10:21.543830 7768 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b48fee3b6474a23a70b62be3d2b7f2f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5afbc257ec7449b189377b1ae0cbebd5" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "5afbc257ec7449b189377b1ae0cbebd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45453 } } peers { permanent_uuid: "8971f3cf560c49ae9dd58727c3b669c4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42317 } attrs { promote: false } } peers { permanent_uuid: "3b48fee3b6474a23a70b62be3d2b7f2f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43103 } attrs { promote: false } } }
I20260926 01:10:21.543905 7768 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b48fee3b6474a23a70b62be3d2b7f2f [sys.catalog]: This master's current role is: FOLLOWER
I20260926 01:10:21.543864 7776 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8971f3cf560c49ae9dd58727c3b669c4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5afbc257ec7449b189377b1ae0cbebd5" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "5afbc257ec7449b189377b1ae0cbebd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45453 } } peers { permanent_uuid: "8971f3cf560c49ae9dd58727c3b669c4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42317 } attrs { promote: false } } peers { permanent_uuid: "3b48fee3b6474a23a70b62be3d2b7f2f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43103 } attrs { promote: false } } }
I20260926 01:10:21.543934 7776 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8971f3cf560c49ae9dd58727c3b669c4 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 01:10:21.559294 7757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:21.557064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53176 (local address 127.0.98.3:43231)
0926 01:10:21.557125 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:21.557127 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:21.557135 (+ 8us) server_negotiation.cc:407] Connection header received
0926 01:10:21.557171 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:21.557172 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:21.557193 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:21.557249 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:10:21.557368 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:21.557901 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:21.558584 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:21.558800 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:21.558855 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:21.558997 (+ 142us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:10:21.559105 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:10:21.559175 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:21.559197 (+ 22us) server_negotiation.cc:299] Negotiation successful
0926 01:10:21.559217 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 01:10:21.561041 7414 raft_consensus.cc:1261] T e12eb2c62b114733ac41963b3d8cb886 P 2264518cb0fe444da3e1606269628249 [term 1 FOLLOWER]: Refusing update from remote peer 72a12d78947e445c85a8d6222cc136af: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:10:21.561046 7551 raft_consensus.cc:1261] T e12eb2c62b114733ac41963b3d8cb886 P 1ffe4fc932934431b944d78dfe891007 [term 1 FOLLOWER]: Refusing update from remote peer 72a12d78947e445c85a8d6222cc136af: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:10:21.561352 7772 consensus_queue.cc:1035] T e12eb2c62b114733ac41963b3d8cb886 P 72a12d78947e445c85a8d6222cc136af [LEADER]: Connected to new peer: Peer: permanent_uuid: "2264518cb0fe444da3e1606269628249" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:10:21.561482 7765 consensus_queue.cc:1035] T e12eb2c62b114733ac41963b3d8cb886 P 72a12d78947e445c85a8d6222cc136af [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ffe4fc932934431b944d78dfe891007" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:10:21.564966 7780 mvcc.cc:205] Tried to move back new op lower bound from 7333417048311607296 to 7333417048225587200. Current Snapshot: MvccSnapshot[applied={T|T < 7333417048311607296 or (T in {7333417048311607296})}]
I20260926 01:10:21.565079 7779 mvcc.cc:205] Tried to move back new op lower bound from 7333417048311607296 to 7333417048225587200. Current Snapshot: MvccSnapshot[applied={T|T < 7333417048311607296}]
I20260926 01:10:21.566689 6490 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39600:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:10:21.566778 6490 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39600:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:10:21.567996 6490 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5afbc257ec7449b189377b1ae0cbebd5: Sending DeleteTablet for 3 replicas of tablet e12eb2c62b114733ac41963b3d8cb886
I20260926 01:10:21.568327 7667 tablet_service.cc:1514] Processing DeleteTablet for tablet e12eb2c62b114733ac41963b3d8cb886 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:10:21 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53170
I20260926 01:10:21.568331 7394 tablet_service.cc:1514] Processing DeleteTablet for tablet e12eb2c62b114733ac41963b3d8cb886 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:10:21 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60344
I20260926 01:10:21.568456 7531 tablet_service.cc:1514] Processing DeleteTablet for tablet e12eb2c62b114733ac41963b3d8cb886 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:10:21 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41022
I20260926 01:10:21.568725 7789 tablet_replica.cc:330] T e12eb2c62b114733ac41963b3d8cb886 P 1ffe4fc932934431b944d78dfe891007: stopping tablet replica
I20260926 01:10:21.568826 7789 raft_consensus.cc:2229] T e12eb2c62b114733ac41963b3d8cb886 P 1ffe4fc932934431b944d78dfe891007 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:10:21.568931 7789 raft_consensus.cc:2258] T e12eb2c62b114733ac41963b3d8cb886 P 1ffe4fc932934431b944d78dfe891007 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:10:21.569335 7787 tablet_replica.cc:330] T e12eb2c62b114733ac41963b3d8cb886 P 2264518cb0fe444da3e1606269628249: stopping tablet replica
I20260926 01:10:21.569433 7787 raft_consensus.cc:2229] T e12eb2c62b114733ac41963b3d8cb886 P 2264518cb0fe444da3e1606269628249 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:10:21.569438 7789 ts_tablet_manager.cc:1905] T e12eb2c62b114733ac41963b3d8cb886 P 1ffe4fc932934431b944d78dfe891007: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:10:21.569540 7787 raft_consensus.cc:2258] T e12eb2c62b114733ac41963b3d8cb886 P 2264518cb0fe444da3e1606269628249 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:10:21.569682 7788 tablet_replica.cc:330] T e12eb2c62b114733ac41963b3d8cb886 P 72a12d78947e445c85a8d6222cc136af: stopping tablet replica
I20260926 01:10:21.569773 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 6459
I20260926 01:10:21.569783 7788 raft_consensus.cc:2229] T e12eb2c62b114733ac41963b3d8cb886 P 72a12d78947e445c85a8d6222cc136af [term 1 LEADER]: Raft consensus shutting down.
I20260926 01:10:21.569895 7788 raft_consensus.cc:2258] T e12eb2c62b114733ac41963b3d8cb886 P 72a12d78947e445c85a8d6222cc136af [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:10:21.570391 7787 ts_tablet_manager.cc:1905] T e12eb2c62b114733ac41963b3d8cb886 P 2264518cb0fe444da3e1606269628249: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:10:21.571221 7789 ts_tablet_manager.cc:1918] T e12eb2c62b114733ac41963b3d8cb886 P 1ffe4fc932934431b944d78dfe891007: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:10:21.571343 7789 log.cc:1196] T e12eb2c62b114733ac41963b3d8cb886 P 1ffe4fc932934431b944d78dfe891007: Deleting WAL directory at /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/ts-1/wal/wals/e12eb2c62b114733ac41963b3d8cb886
I20260926 01:10:21.571653 7789 ts_tablet_manager.cc:1939] T e12eb2c62b114733ac41963b3d8cb886 P 1ffe4fc932934431b944d78dfe891007: Deleting consensus metadata
I20260926 01:10:21.571858 7787 ts_tablet_manager.cc:1918] T e12eb2c62b114733ac41963b3d8cb886 P 2264518cb0fe444da3e1606269628249: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:10:21.571928 7787 log.cc:1196] T e12eb2c62b114733ac41963b3d8cb886 P 2264518cb0fe444da3e1606269628249: Deleting WAL directory at /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/ts-0/wal/wals/e12eb2c62b114733ac41963b3d8cb886
I20260926 01:10:21.572151 7787 ts_tablet_manager.cc:1939] T e12eb2c62b114733ac41963b3d8cb886 P 2264518cb0fe444da3e1606269628249: Deleting consensus metadata
I20260926 01:10:21.572751 7788 ts_tablet_manager.cc:1905] T e12eb2c62b114733ac41963b3d8cb886 P 72a12d78947e445c85a8d6222cc136af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:10:21.574291 7788 ts_tablet_manager.cc:1918] T e12eb2c62b114733ac41963b3d8cb886 P 72a12d78947e445c85a8d6222cc136af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:10:21.574366 7788 log.cc:1196] T e12eb2c62b114733ac41963b3d8cb886 P 72a12d78947e445c85a8d6222cc136af: Deleting WAL directory at /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/ts-2/wal/wals/e12eb2c62b114733ac41963b3d8cb886
I20260926 01:10:21.574609 7788 ts_tablet_manager.cc:1939] T e12eb2c62b114733ac41963b3d8cb886 P 72a12d78947e445c85a8d6222cc136af: Deleting consensus metadata
W20260926 01:10:21.575663 7487 connection.cc:535] server connection from 127.0.0.1:41022 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260926 01:10:21.575677 7350 connection.cc:535] server connection from 127.0.0.1:60344 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260926 01:10:21.575904 7620 connection.cc:535] server connection from 127.0.0.1:53170 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 01:10:21.576678 7746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:21.568961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49124 (local address 127.0.98.60:43103)
0926 01:10:21.569000 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:21.569001 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:21.569009 (+ 8us) server_negotiation.cc:407] Connection header received
0926 01:10:21.569029 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:21.569029 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:21.569038 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:21.569088 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:21.570240 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:21.570957 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:21.572192 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:21.572492 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:21.572560 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:21.574116 (+ 1556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:10:21.574117 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:10:21.574117 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:10:21.574125 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:10:21.575608 (+ 1483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:21.575973 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:21.575975 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:21.575975 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:21.576029 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:21.576211 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:21.576212 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:21.576212 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:21.576325 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:10:21.576407 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:21.576516 (+ 109us) server_negotiation.cc:299] Negotiation successful
0926 01:10:21.576591 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 01:10:21.584193 7749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:21.577740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32854 (local address 127.0.98.61:42317)
0926 01:10:21.577780 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:21.577782 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:21.577924 (+ 142us) server_negotiation.cc:407] Connection header received
0926 01:10:21.577979 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:21.577980 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:21.577993 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:21.578043 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:21.578613 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:21.579342 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:21.580312 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:21.580614 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:21.580694 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:21.581897 (+ 1203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:10:21.581898 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:10:21.581899 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:10:21.581906 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:10:21.583150 (+ 1244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:21.583511 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:21.583512 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:21.583512 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:21.583564 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:21.583729 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:21.583730 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:21.583730 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:21.583838 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:10:21.583921 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:21.584011 (+ 90us) server_negotiation.cc:299] Negotiation successful
0926 01:10:21.584092 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 01:10:21.584342 7746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:21.577455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49128 (local address 127.0.98.60:43103)
0926 01:10:21.577497 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:21.577499 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:21.577561 (+ 62us) server_negotiation.cc:407] Connection header received
0926 01:10:21.577638 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:21.577639 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:21.577649 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:21.577677 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:21.578467 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:21.578993 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:21.579803 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:21.580093 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:21.580145 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:21.581599 (+ 1454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:10:21.581600 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:10:21.581600 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:10:21.581607 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:10:21.583104 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:21.583562 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:21.583563 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:21.583564 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:21.583619 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:21.583800 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:21.583801 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:21.583802 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:21.583918 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:10:21.583985 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:21.584177 (+ 192us) server_negotiation.cc:299] Negotiation successful
0926 01:10:21.584251 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 01:10:22.565605 7797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:22.565367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45453 (local address 127.0.98.3:37505)
0926 01:10:22.565478 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:22.565555 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":64,"threads_started":1}
I20260926 01:10:22.565603 7798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:22.565367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45453 (local address 127.0.98.2:39915)
0926 01:10:22.565478 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:22.565553 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":64,"threads_started":1}
W20260926 01:10:22.565929 7734 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45453 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:45453: connect: Connection refused (error 111)
W20260926 01:10:22.565960 7598 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45453 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:45453: connect: Connection refused (error 111)
I20260926 01:10:22.567050 7799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:22.566820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45453 (local address 127.0.98.1:56029)
0926 01:10:22.566932 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:22.567004 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":51,"threads_started":1}
W20260926 01:10:22.567248 7461 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45453 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:45453: connect: Connection refused (error 111)
I20260926 01:10:23.068552 7800 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8971f3cf560c49ae9dd58727c3b669c4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5afbc257ec7449b189377b1ae0cbebd5)
I20260926 01:10:23.068632 7801 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3b48fee3b6474a23a70b62be3d2b7f2f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5afbc257ec7449b189377b1ae0cbebd5)
I20260926 01:10:23.068717 7801 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3b48fee3b6474a23a70b62be3d2b7f2f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "5afbc257ec7449b189377b1ae0cbebd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45453 } } peers { permanent_uuid: "8971f3cf560c49ae9dd58727c3b669c4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42317 } attrs { promote: false } } peers { permanent_uuid: "3b48fee3b6474a23a70b62be3d2b7f2f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43103 } attrs { promote: false } }
I20260926 01:10:23.068706 7800 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8971f3cf560c49ae9dd58727c3b669c4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "5afbc257ec7449b189377b1ae0cbebd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45453 } } peers { permanent_uuid: "8971f3cf560c49ae9dd58727c3b669c4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42317 } attrs { promote: false } } peers { permanent_uuid: "3b48fee3b6474a23a70b62be3d2b7f2f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43103 } attrs { promote: false } }
I20260926 01:10:23.068899 7801 leader_election.cc:288] T 00000000000000000000000000000000 P 3b48fee3b6474a23a70b62be3d2b7f2f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5afbc257ec7449b189377b1ae0cbebd5 (127.0.98.62:45453), 8971f3cf560c49ae9dd58727c3b669c4 (127.0.98.61:42317)
I20260926 01:10:23.069291 7802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:23.069044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45453 (local address 127.0.0.1:40078)
0926 01:10:23.069190 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:23.069249 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":44,"threads_started":1}
I20260926 01:10:23.069379 7800 leader_election.cc:288] T 00000000000000000000000000000000 P 8971f3cf560c49ae9dd58727c3b669c4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5afbc257ec7449b189377b1ae0cbebd5 (127.0.98.62:45453), 3b48fee3b6474a23a70b62be3d2b7f2f (127.0.98.60:43103)
W20260926 01:10:23.069574 7267 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45453: connect: Connection refused (error 111)
I20260926 01:10:23.069626 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:23.069073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45453 (local address 127.0.0.1:40090)
0926 01:10:23.069554 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:23.069595 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":51,"threads_started":1}
W20260926 01:10:23.069711 7074 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45453: connect: Connection refused (error 111)
I20260926 01:10:23.070101 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:23.070025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45453 (local address 127.0.0.1:40100)
0926 01:10:23.070057 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:23.070078 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260926 01:10:23.070133 7805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:23.070039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45453 (local address 127.0.0.1:40114)
0926 01:10:23.070074 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:23.070106 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260926 01:10:23.070166 7074 leader_election.cc:334] T 00000000000000000000000000000000 P 8971f3cf560c49ae9dd58727c3b669c4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5afbc257ec7449b189377b1ae0cbebd5 (127.0.98.62:45453): Network error: Client connection negotiation failed: client connection to 127.0.98.62:45453: connect: Connection refused (error 111)
W20260926 01:10:23.070282 7267 leader_election.cc:334] T 00000000000000000000000000000000 P 3b48fee3b6474a23a70b62be3d2b7f2f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5afbc257ec7449b189377b1ae0cbebd5 (127.0.98.62:45453): Network error: Client connection negotiation failed: client connection to 127.0.98.62:45453: connect: Connection refused (error 111)
I20260926 01:10:23.072557 7802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:23.069176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42317 (local address 127.0.0.1:32856)
0926 01:10:23.069357 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:23.069371 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:23.069432 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:23.069660 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:23.069661 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:23.069663 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:10:23.069725 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:23.069726 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:23.070602 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:23.070603 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:23.071788 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:23.071791 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:23.072329 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:23.072330 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:23.072400 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:23.072404 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 01:10:23.072450 (+ 46us) client_negotiation.cc:240] Negotiation successful
0926 01:10:23.072491 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":36,"threads_started":1}
I20260926 01:10:23.072557 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:23.069307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32856 (local address 127.0.98.61:42317)
0926 01:10:23.069526 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:23.069529 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:23.069536 (+ 7us) server_negotiation.cc:407] Connection header received
0926 01:10:23.069558 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:23.069559 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:23.069568 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:23.069785 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:10:23.069841 (+ 56us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:23.070532 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:23.071873 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:23.072241 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:23.072310 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:23.072323 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:23.072462 (+ 139us) server_negotiation.cc:299] Negotiation successful
0926 01:10:23.072497 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":42,"threads_started":1}
I20260926 01:10:23.072654 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:23.069315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49142 (local address 127.0.98.60:43103)
0926 01:10:23.069508 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:23.069510 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:23.069767 (+ 257us) server_negotiation.cc:407] Connection header received
0926 01:10:23.069902 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:23.069903 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:23.069913 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:23.069957 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:10:23.070142 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:23.070706 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:23.072004 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:23.072344 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:23.072406 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:23.072415 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:23.072531 (+ 116us) server_negotiation.cc:299] Negotiation successful
0926 01:10:23.072568 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":40,"threads_started":1}
I20260926 01:10:23.072667 7804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:23.069129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43103 (local address 127.0.0.1:49142)
0926 01:10:23.069713 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:23.069722 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:23.069791 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:23.070021 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:23.070022 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:23.070024 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:10:23.070069 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:23.070071 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:23.070795 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:23.070795 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:23.071938 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:23.071940 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:23.072411 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:23.072412 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:23.072474 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:23.072478 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 01:10:23.072522 (+ 44us) client_negotiation.cc:240] Negotiation successful
0926 01:10:23.072568 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":46,"threads_started":1}
I20260926 01:10:23.072755 7098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3b48fee3b6474a23a70b62be3d2b7f2f" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "8971f3cf560c49ae9dd58727c3b669c4" is_pre_election: true
I20260926 01:10:23.072840 7098 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8971f3cf560c49ae9dd58727c3b669c4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b48fee3b6474a23a70b62be3d2b7f2f in term 1.
I20260926 01:10:23.072943 7291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8971f3cf560c49ae9dd58727c3b669c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "3b48fee3b6474a23a70b62be3d2b7f2f" is_pre_election: true
I20260926 01:10:23.072979 7267 leader_election.cc:302] T 00000000000000000000000000000000 P 3b48fee3b6474a23a70b62be3d2b7f2f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b48fee3b6474a23a70b62be3d2b7f2f, 8971f3cf560c49ae9dd58727c3b669c4; no voters: 5afbc257ec7449b189377b1ae0cbebd5
I20260926 01:10:23.073042 7291 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3b48fee3b6474a23a70b62be3d2b7f2f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8971f3cf560c49ae9dd58727c3b669c4 in term 1.
I20260926 01:10:23.073096 7801 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3b48fee3b6474a23a70b62be3d2b7f2f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260926 01:10:23.073143 7801 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3b48fee3b6474a23a70b62be3d2b7f2f [term 1 FOLLOWER]: Starting leader election (detected failure of leader 5afbc257ec7449b189377b1ae0cbebd5)
I20260926 01:10:23.073161 7801 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3b48fee3b6474a23a70b62be3d2b7f2f [term 1 FOLLOWER]: Advancing to term 2
I20260926 01:10:23.073173 7073 leader_election.cc:302] T 00000000000000000000000000000000 P 8971f3cf560c49ae9dd58727c3b669c4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b48fee3b6474a23a70b62be3d2b7f2f, 8971f3cf560c49ae9dd58727c3b669c4; no voters: 5afbc257ec7449b189377b1ae0cbebd5
I20260926 01:10:23.073271 7800 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8971f3cf560c49ae9dd58727c3b669c4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260926 01:10:23.073313 7800 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8971f3cf560c49ae9dd58727c3b669c4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 5afbc257ec7449b189377b1ae0cbebd5)
I20260926 01:10:23.073338 7800 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8971f3cf560c49ae9dd58727c3b669c4 [term 1 FOLLOWER]: Advancing to term 2
I20260926 01:10:23.073910 7801 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3b48fee3b6474a23a70b62be3d2b7f2f [term 2 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "5afbc257ec7449b189377b1ae0cbebd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45453 } } peers { permanent_uuid: "8971f3cf560c49ae9dd58727c3b669c4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42317 } attrs { promote: false } } peers { permanent_uuid: "3b48fee3b6474a23a70b62be3d2b7f2f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43103 } attrs { promote: false } }
I20260926 01:10:23.074023 7801 leader_election.cc:288] T 00000000000000000000000000000000 P 3b48fee3b6474a23a70b62be3d2b7f2f [CANDIDATE]: Term 2 election: Requested vote from peers 5afbc257ec7449b189377b1ae0cbebd5 (127.0.98.62:45453), 8971f3cf560c49ae9dd58727c3b669c4 (127.0.98.61:42317)
I20260926 01:10:23.073997 7800 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8971f3cf560c49ae9dd58727c3b669c4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "5afbc257ec7449b189377b1ae0cbebd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45453 } } peers { permanent_uuid: "8971f3cf560c49ae9dd58727c3b669c4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42317 } attrs { promote: false } } peers { permanent_uuid: "3b48fee3b6474a23a70b62be3d2b7f2f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43103 } attrs { promote: false } }
I20260926 01:10:23.074119 7800 leader_election.cc:288] T 00000000000000000000000000000000 P 8971f3cf560c49ae9dd58727c3b669c4 [CANDIDATE]: Term 2 election: Requested vote from peers 5afbc257ec7449b189377b1ae0cbebd5 (127.0.98.62:45453), 3b48fee3b6474a23a70b62be3d2b7f2f (127.0.98.60:43103)
I20260926 01:10:23.074265 7802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:23.074149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45453 (local address 127.0.0.1:40124)
0926 01:10:23.074219 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:23.074243 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260926 01:10:23.074249 7804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:23.074165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45453 (local address 127.0.0.1:40138)
0926 01:10:23.074198 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:23.074227 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260926 01:10:23.074247 7098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3b48fee3b6474a23a70b62be3d2b7f2f" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "8971f3cf560c49ae9dd58727c3b669c4"
I20260926 01:10:23.074327 7098 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 8971f3cf560c49ae9dd58727c3b669c4 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b48fee3b6474a23a70b62be3d2b7f2f in current term 2: Already voted for candidate 8971f3cf560c49ae9dd58727c3b669c4 in this term.
I20260926 01:10:23.074390 7291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8971f3cf560c49ae9dd58727c3b669c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "3b48fee3b6474a23a70b62be3d2b7f2f"
I20260926 01:10:23.074457 7291 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 3b48fee3b6474a23a70b62be3d2b7f2f [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 8971f3cf560c49ae9dd58727c3b669c4 in current term 2: Already voted for candidate 3b48fee3b6474a23a70b62be3d2b7f2f in this term.
I20260926 01:10:23.074662 7802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:23.074578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45453 (local address 127.0.0.1:40146)
0926 01:10:23.074610 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:23.074638 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
W20260926 01:10:23.074735 7267 leader_election.cc:334] T 00000000000000000000000000000000 P 3b48fee3b6474a23a70b62be3d2b7f2f [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 5afbc257ec7449b189377b1ae0cbebd5 (127.0.98.62:45453): Network error: Client connection negotiation failed: client connection to 127.0.98.62:45453: connect: Connection refused (error 111)
I20260926 01:10:23.074769 7804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:23.074579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45453 (local address 127.0.0.1:40152)
0926 01:10:23.074733 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:23.074753 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142}
I20260926 01:10:23.074807 7267 leader_election.cc:302] T 00000000000000000000000000000000 P 3b48fee3b6474a23a70b62be3d2b7f2f [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b48fee3b6474a23a70b62be3d2b7f2f; no voters: 5afbc257ec7449b189377b1ae0cbebd5, 8971f3cf560c49ae9dd58727c3b669c4
W20260926 01:10:23.074826 7074 leader_election.cc:334] T 00000000000000000000000000000000 P 8971f3cf560c49ae9dd58727c3b669c4 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 5afbc257ec7449b189377b1ae0cbebd5 (127.0.98.62:45453): Network error: Client connection negotiation failed: client connection to 127.0.98.62:45453: connect: Connection refused (error 111)
I20260926 01:10:23.074865 7074 leader_election.cc:302] T 00000000000000000000000000000000 P 8971f3cf560c49ae9dd58727c3b669c4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8971f3cf560c49ae9dd58727c3b669c4; no voters: 3b48fee3b6474a23a70b62be3d2b7f2f, 5afbc257ec7449b189377b1ae0cbebd5
I20260926 01:10:23.074908 7801 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 3b48fee3b6474a23a70b62be3d2b7f2f [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260926 01:10:23.074956 7800 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 8971f3cf560c49ae9dd58727c3b669c4 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260926 01:10:23.566556 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:23.566321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45453 (local address 127.0.98.3:36113)
0926 01:10:23.566472 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:23.566505 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":82,"threads_started":1}
I20260926 01:10:23.566556 7811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:23.566338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45453 (local address 127.0.98.2:32819)
0926 01:10:23.566464 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:23.566505 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":62,"threads_started":1}
I20260926 01:10:23.567662 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:23.567480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45453 (local address 127.0.98.1:48623)
0926 01:10:23.567581 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:23.567619 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":62,"threads_started":1}
I20260926 01:10:24.567167 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:24.566975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45453 (local address 127.0.98.3:56989)
0926 01:10:24.567084 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:24.567122 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":66,"threads_started":1}
I20260926 01:10:24.567176 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:24.566962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45453 (local address 127.0.98.2:35689)
0926 01:10:24.567098 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:24.567134 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":63,"threads_started":1}
I20260926 01:10:24.568189 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:24.568003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45453 (local address 127.0.98.1:37521)
0926 01:10:24.568131 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:24.568155 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":55,"threads_started":1}
I20260926 01:10:24.892992 7816 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3b48fee3b6474a23a70b62be3d2b7f2f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:10:24.893087 7816 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3b48fee3b6474a23a70b62be3d2b7f2f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "5afbc257ec7449b189377b1ae0cbebd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45453 } } peers { permanent_uuid: "8971f3cf560c49ae9dd58727c3b669c4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42317 } attrs { promote: false } } peers { permanent_uuid: "3b48fee3b6474a23a70b62be3d2b7f2f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43103 } attrs { promote: false } }
I20260926 01:10:24.893492 7816 leader_election.cc:288] T 00000000000000000000000000000000 P 3b48fee3b6474a23a70b62be3d2b7f2f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5afbc257ec7449b189377b1ae0cbebd5 (127.0.98.62:45453), 8971f3cf560c49ae9dd58727c3b669c4 (127.0.98.61:42317)
I20260926 01:10:24.893584 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:24.893381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45453 (local address 127.0.0.1:40418)
0926 01:10:24.893487 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:24.893537 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":58,"threads_started":1}
I20260926 01:10:24.893736 7098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3b48fee3b6474a23a70b62be3d2b7f2f" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "8971f3cf560c49ae9dd58727c3b669c4" is_pre_election: true
I20260926 01:10:24.893872 7098 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8971f3cf560c49ae9dd58727c3b669c4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b48fee3b6474a23a70b62be3d2b7f2f in term 2.
I20260926 01:10:24.894016 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:24.893923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45453 (local address 127.0.0.1:40428)
0926 01:10:24.893957 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:24.893989 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
I20260926 01:10:24.894037 7267 leader_election.cc:302] T 00000000000000000000000000000000 P 3b48fee3b6474a23a70b62be3d2b7f2f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b48fee3b6474a23a70b62be3d2b7f2f, 8971f3cf560c49ae9dd58727c3b669c4; no voters:
W20260926 01:10:24.894101 7267 leader_election.cc:334] T 00000000000000000000000000000000 P 3b48fee3b6474a23a70b62be3d2b7f2f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 5afbc257ec7449b189377b1ae0cbebd5 (127.0.98.62:45453): Network error: Client connection negotiation failed: client connection to 127.0.98.62:45453: connect: Connection refused (error 111)
I20260926 01:10:24.894107 7816 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3b48fee3b6474a23a70b62be3d2b7f2f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260926 01:10:24.894150 7816 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3b48fee3b6474a23a70b62be3d2b7f2f [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:10:24.894166 7816 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3b48fee3b6474a23a70b62be3d2b7f2f [term 2 FOLLOWER]: Advancing to term 3
I20260926 01:10:24.894834 7816 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3b48fee3b6474a23a70b62be3d2b7f2f [term 3 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "5afbc257ec7449b189377b1ae0cbebd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45453 } } peers { permanent_uuid: "8971f3cf560c49ae9dd58727c3b669c4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42317 } attrs { promote: false } } peers { permanent_uuid: "3b48fee3b6474a23a70b62be3d2b7f2f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43103 } attrs { promote: false } }
I20260926 01:10:24.894936 7816 leader_election.cc:288] T 00000000000000000000000000000000 P 3b48fee3b6474a23a70b62be3d2b7f2f [CANDIDATE]: Term 3 election: Requested vote from peers 5afbc257ec7449b189377b1ae0cbebd5 (127.0.98.62:45453), 8971f3cf560c49ae9dd58727c3b669c4 (127.0.98.61:42317)
I20260926 01:10:24.895105 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:24.895013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45453 (local address 127.0.0.1:40436)
0926 01:10:24.895056 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:24.895085 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260926 01:10:24.895159 7098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3b48fee3b6474a23a70b62be3d2b7f2f" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "8971f3cf560c49ae9dd58727c3b669c4"
I20260926 01:10:24.895232 7098 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8971f3cf560c49ae9dd58727c3b669c4 [term 2 FOLLOWER]: Advancing to term 3
I20260926 01:10:24.895380 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:24.895315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45453 (local address 127.0.0.1:40448)
0926 01:10:24.895342 (+ 27us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:24.895365 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260926 01:10:24.895447 7267 leader_election.cc:334] T 00000000000000000000000000000000 P 3b48fee3b6474a23a70b62be3d2b7f2f [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 5afbc257ec7449b189377b1ae0cbebd5 (127.0.98.62:45453): Network error: Client connection negotiation failed: client connection to 127.0.98.62:45453: connect: Connection refused (error 111)
I20260926 01:10:24.895880 7098 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8971f3cf560c49ae9dd58727c3b669c4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b48fee3b6474a23a70b62be3d2b7f2f in term 3.
I20260926 01:10:24.896035 7267 leader_election.cc:302] T 00000000000000000000000000000000 P 3b48fee3b6474a23a70b62be3d2b7f2f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b48fee3b6474a23a70b62be3d2b7f2f, 8971f3cf560c49ae9dd58727c3b669c4; no voters: 5afbc257ec7449b189377b1ae0cbebd5
I20260926 01:10:24.896126 7816 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3b48fee3b6474a23a70b62be3d2b7f2f [term 3 FOLLOWER]: Leader election won for term 3
I20260926 01:10:24.896247 7816 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3b48fee3b6474a23a70b62be3d2b7f2f [term 3 LEADER]: Becoming Leader. State: Replica: 3b48fee3b6474a23a70b62be3d2b7f2f, State: Running, Role: LEADER
I20260926 01:10:24.896335 7816 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3b48fee3b6474a23a70b62be3d2b7f2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "5afbc257ec7449b189377b1ae0cbebd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45453 } } peers { permanent_uuid: "8971f3cf560c49ae9dd58727c3b669c4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42317 } attrs { promote: false } } peers { permanent_uuid: "3b48fee3b6474a23a70b62be3d2b7f2f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43103 } attrs { promote: false } }
I20260926 01:10:24.896657 7819 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b48fee3b6474a23a70b62be3d2b7f2f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b48fee3b6474a23a70b62be3d2b7f2f. Latest consensus state: current_term: 3 leader_uuid: "3b48fee3b6474a23a70b62be3d2b7f2f" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "5afbc257ec7449b189377b1ae0cbebd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45453 } } peers { permanent_uuid: "8971f3cf560c49ae9dd58727c3b669c4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42317 } attrs { promote: false } } peers { permanent_uuid: "3b48fee3b6474a23a70b62be3d2b7f2f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43103 } attrs { promote: false } } }
I20260926 01:10:24.896749 7819 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b48fee3b6474a23a70b62be3d2b7f2f [sys.catalog]: This master's current role is: LEADER
I20260926 01:10:24.896914 7821 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:10:24.897287 7821 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:10:24.897337 7821 catalog_manager.cc:1259] Loaded cluster ID: d16d612222b24a05bd8871dd9f254e18
I20260926 01:10:24.897357 7821 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 01:10:24.897501 7821 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:10:24.897552 7821 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3b48fee3b6474a23a70b62be3d2b7f2f: Loaded TSK: 0
I20260926 01:10:24.897646 7821 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 01:10:24.907744 7280 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49128:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:10:24.909293 7280 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:10:24.910198 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:24.910094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45453 (local address 127.0.0.1:40458)
0926 01:10:24.910136 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:24.910167 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260926 01:10:24.910305 7098 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8971f3cf560c49ae9dd58727c3b669c4 [term 3 FOLLOWER]: Refusing update from remote peer 3b48fee3b6474a23a70b62be3d2b7f2f: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260926 01:10:24.910478 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:24.910411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45453 (local address 127.0.0.1:40472)
0926 01:10:24.910435 (+ 24us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:24.910460 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":13}
I20260926 01:10:24.910523 7816 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3b48fee3b6474a23a70b62be3d2b7f2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8971f3cf560c49ae9dd58727c3b669c4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42317 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260926 01:10:24.910691 7267 consensus_peers.cc:487] T 00000000000000000000000000000000 P 3b48fee3b6474a23a70b62be3d2b7f2f -> Peer 5afbc257ec7449b189377b1ae0cbebd5 (127.0.98.62:45453): Couldn't send request to peer 5afbc257ec7449b189377b1ae0cbebd5. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45453: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260926 01:10:24.911119 7823 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8971f3cf560c49ae9dd58727c3b669c4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b48fee3b6474a23a70b62be3d2b7f2f. Latest consensus state: current_term: 3 leader_uuid: "3b48fee3b6474a23a70b62be3d2b7f2f" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "5afbc257ec7449b189377b1ae0cbebd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45453 } } peers { permanent_uuid: "8971f3cf560c49ae9dd58727c3b669c4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42317 } attrs { promote: false } } peers { permanent_uuid: "3b48fee3b6474a23a70b62be3d2b7f2f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43103 } attrs { promote: false } } }
I20260926 01:10:24.911196 7823 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8971f3cf560c49ae9dd58727c3b669c4 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 01:10:24.911712 7819 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b48fee3b6474a23a70b62be3d2b7f2f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "3b48fee3b6474a23a70b62be3d2b7f2f" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "5afbc257ec7449b189377b1ae0cbebd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45453 } } peers { permanent_uuid: "8971f3cf560c49ae9dd58727c3b669c4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42317 } attrs { promote: false } } peers { permanent_uuid: "3b48fee3b6474a23a70b62be3d2b7f2f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43103 } attrs { promote: false } } }
I20260926 01:10:24.911795 7819 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b48fee3b6474a23a70b62be3d2b7f2f [sys.catalog]: This master's current role is: LEADER
I20260926 01:10:24.912232 7823 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8971f3cf560c49ae9dd58727c3b669c4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "3b48fee3b6474a23a70b62be3d2b7f2f" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "5afbc257ec7449b189377b1ae0cbebd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45453 } } peers { permanent_uuid: "8971f3cf560c49ae9dd58727c3b669c4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42317 } attrs { promote: false } } peers { permanent_uuid: "3b48fee3b6474a23a70b62be3d2b7f2f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43103 } attrs { promote: false } } }
I20260926 01:10:24.912313 7823 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8971f3cf560c49ae9dd58727c3b669c4 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 01:10:24.916054 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:24.913263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39297 (local address 127.0.0.1:41024)
0926 01:10:24.913298 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:24.913316 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:24.913361 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:24.913586 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:24.913587 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:24.913589 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:10:24.913640 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:24.913641 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:24.914374 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:24.914375 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:24.915124 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:24.915126 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:24.915849 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:24.915850 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:24.915914 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:24.915918 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 01:10:24.915944 (+ 26us) client_negotiation.cc:240] Negotiation successful
0926 01:10:24.915984 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21}
I20260926 01:10:24.916473 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:24.913393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35643 (local address 127.0.0.1:60358)
0926 01:10:24.913652 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:24.913663 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:24.913705 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:24.914042 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:24.914043 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:24.914045 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:10:24.914084 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:24.914086 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:24.914747 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:24.914748 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:24.915661 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:24.915663 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:24.916242 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:24.916243 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:24.916306 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:24.916310 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 01:10:24.916354 (+ 44us) client_negotiation.cc:240] Negotiation successful
0926 01:10:24.916394 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":74,"threads_started":1}
I20260926 01:10:24.916508 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:24.913338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41024 (local address 127.0.98.2:39297)
0926 01:10:24.913451 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:24.913453 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:24.913462 (+ 9us) server_negotiation.cc:407] Connection header received
0926 01:10:24.913497 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:24.913498 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:24.913509 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:24.913585 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:10:24.913708 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:24.914298 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:24.915197 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:24.915755 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:24.916400 (+ 645us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:24.916414 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:24.916430 (+ 16us) server_negotiation.cc:299] Negotiation successful
0926 01:10:24.916460 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87,"thread_start_us":56,"threads_started":1}
I20260926 01:10:24.916733 7531 tablet_service.cc:1467] Processing CreateTablet for tablet 2a5a61fa10f74255b736ba2ccacd5d54 (DEFAULT_TABLE table=test-table [id=2efcbd4e061449c7b32691ce889a8ef1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:10:24.916759 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:24.913481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60358 (local address 127.0.98.1:35643)
0926 01:10:24.913946 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:24.913948 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:24.913955 (+ 7us) server_negotiation.cc:407] Connection header received
0926 01:10:24.913978 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:24.913979 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:24.913988 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:24.914040 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:10:24.914131 (+ 91us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:24.914663 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:24.915746 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:24.916171 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:24.916229 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:24.916240 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:24.916643 (+ 403us) server_negotiation.cc:299] Negotiation successful
0926 01:10:24.916691 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":51,"threads_started":1}
I20260926 01:10:24.916870 7531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a5a61fa10f74255b736ba2ccacd5d54. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:10:24.917207 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:24.913581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53188 (local address 127.0.98.3:43231)
0926 01:10:24.913773 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:24.913775 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:24.913782 (+ 7us) server_negotiation.cc:407] Connection header received
0926 01:10:24.913806 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:24.913806 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:24.913848 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:24.913900 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:10:24.914505 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:24.915185 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:24.916356 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:24.916946 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:24.917000 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:24.917010 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:24.917130 (+ 120us) server_negotiation.cc:299] Negotiation successful
0926 01:10:24.917157 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":50,"threads_started":1}
I20260926 01:10:24.917237 7831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:24.913336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43231 (local address 127.0.0.1:53188)
0926 01:10:24.913500 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:24.913508 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:24.913549 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:24.914385 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:24.914386 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:24.914388 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:10:24.914430 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:24.914431 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:24.915258 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:24.915259 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:24.916281 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:24.916283 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:24.917006 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:24.917007 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:24.917063 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:24.917066 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 01:10:24.917125 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 01:10:24.917161 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":64,"threads_started":1}
I20260926 01:10:24.917429 7667 tablet_service.cc:1467] Processing CreateTablet for tablet 2a5a61fa10f74255b736ba2ccacd5d54 (DEFAULT_TABLE table=test-table [id=2efcbd4e061449c7b32691ce889a8ef1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:10:24.917531 7667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a5a61fa10f74255b736ba2ccacd5d54. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:10:24.918183 7834 tablet_bootstrap.cc:492] T 2a5a61fa10f74255b736ba2ccacd5d54 P 1ffe4fc932934431b944d78dfe891007: Bootstrap starting.
I20260926 01:10:24.918516 7834 tablet_bootstrap.cc:654] T 2a5a61fa10f74255b736ba2ccacd5d54 P 1ffe4fc932934431b944d78dfe891007: Neither blocks nor log segments found. Creating new log.
I20260926 01:10:24.918685 7835 tablet_bootstrap.cc:492] T 2a5a61fa10f74255b736ba2ccacd5d54 P 72a12d78947e445c85a8d6222cc136af: Bootstrap starting.
I20260926 01:10:24.918735 7394 tablet_service.cc:1467] Processing CreateTablet for tablet 2a5a61fa10f74255b736ba2ccacd5d54 (DEFAULT_TABLE table=test-table [id=2efcbd4e061449c7b32691ce889a8ef1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:10:24.918838 7394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a5a61fa10f74255b736ba2ccacd5d54. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:10:24.919008 7834 tablet_bootstrap.cc:492] T 2a5a61fa10f74255b736ba2ccacd5d54 P 1ffe4fc932934431b944d78dfe891007: No bootstrap required, opened a new log
I20260926 01:10:24.919009 7835 tablet_bootstrap.cc:654] T 2a5a61fa10f74255b736ba2ccacd5d54 P 72a12d78947e445c85a8d6222cc136af: Neither blocks nor log segments found. Creating new log.
I20260926 01:10:24.919040 7834 ts_tablet_manager.cc:1397] T 2a5a61fa10f74255b736ba2ccacd5d54 P 1ffe4fc932934431b944d78dfe891007: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 01:10:24.919154 7834 raft_consensus.cc:348] T 2a5a61fa10f74255b736ba2ccacd5d54 P 1ffe4fc932934431b944d78dfe891007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ffe4fc932934431b944d78dfe891007" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39297 } } peers { permanent_uuid: "72a12d78947e445c85a8d6222cc136af" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43231 } } peers { permanent_uuid: "2264518cb0fe444da3e1606269628249" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35643 } }
I20260926 01:10:24.919205 7834 raft_consensus.cc:374] T 2a5a61fa10f74255b736ba2ccacd5d54 P 1ffe4fc932934431b944d78dfe891007 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:10:24.919222 7834 raft_consensus.cc:729] T 2a5a61fa10f74255b736ba2ccacd5d54 P 1ffe4fc932934431b944d78dfe891007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ffe4fc932934431b944d78dfe891007, State: Initialized, Role: FOLLOWER
I20260926 01:10:24.919265 7834 consensus_queue.cc:260] T 2a5a61fa10f74255b736ba2ccacd5d54 P 1ffe4fc932934431b944d78dfe891007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ffe4fc932934431b944d78dfe891007" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39297 } } peers { permanent_uuid: "72a12d78947e445c85a8d6222cc136af" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43231 } } peers { permanent_uuid: "2264518cb0fe444da3e1606269628249" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35643 } }
I20260926 01:10:24.919427 7834 ts_tablet_manager.cc:1428] T 2a5a61fa10f74255b736ba2ccacd5d54 P 1ffe4fc932934431b944d78dfe891007: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:24.919725 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:24.919635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45453 (local address 127.0.98.2:56045)
0926 01:10:24.919670 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:24.919698 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260926 01:10:24.920116 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:24.919972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45453 (local address 127.0.98.2:48461)
0926 01:10:24.920064 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:24.920090 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260926 01:10:24.920320 7835 tablet_bootstrap.cc:492] T 2a5a61fa10f74255b736ba2ccacd5d54 P 72a12d78947e445c85a8d6222cc136af: No bootstrap required, opened a new log
I20260926 01:10:24.920375 7835 ts_tablet_manager.cc:1397] T 2a5a61fa10f74255b736ba2ccacd5d54 P 72a12d78947e445c85a8d6222cc136af: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 01:10:24.920485 7835 raft_consensus.cc:348] T 2a5a61fa10f74255b736ba2ccacd5d54 P 72a12d78947e445c85a8d6222cc136af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ffe4fc932934431b944d78dfe891007" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39297 } } peers { permanent_uuid: "72a12d78947e445c85a8d6222cc136af" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43231 } } peers { permanent_uuid: "2264518cb0fe444da3e1606269628249" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35643 } }
I20260926 01:10:24.920544 7839 tablet_bootstrap.cc:492] T 2a5a61fa10f74255b736ba2ccacd5d54 P 2264518cb0fe444da3e1606269628249: Bootstrap starting.
I20260926 01:10:24.920578 7835 raft_consensus.cc:374] T 2a5a61fa10f74255b736ba2ccacd5d54 P 72a12d78947e445c85a8d6222cc136af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:10:24.920598 7835 raft_consensus.cc:729] T 2a5a61fa10f74255b736ba2ccacd5d54 P 72a12d78947e445c85a8d6222cc136af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72a12d78947e445c85a8d6222cc136af, State: Initialized, Role: FOLLOWER
I20260926 01:10:24.920652 7835 consensus_queue.cc:260] T 2a5a61fa10f74255b736ba2ccacd5d54 P 72a12d78947e445c85a8d6222cc136af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ffe4fc932934431b944d78dfe891007" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39297 } } peers { permanent_uuid: "72a12d78947e445c85a8d6222cc136af" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43231 } } peers { permanent_uuid: "2264518cb0fe444da3e1606269628249" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35643 } }
I20260926 01:10:24.920818 7835 ts_tablet_manager.cc:1428] T 2a5a61fa10f74255b736ba2ccacd5d54 P 72a12d78947e445c85a8d6222cc136af: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:24.920885 7839 tablet_bootstrap.cc:654] T 2a5a61fa10f74255b736ba2ccacd5d54 P 2264518cb0fe444da3e1606269628249: Neither blocks nor log segments found. Creating new log.
I20260926 01:10:24.921195 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:24.921106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45453 (local address 127.0.98.3:56713)
0926 01:10:24.921141 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:24.921167 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260926 01:10:24.921397 7733 heartbeater.cc:503] Master 127.0.98.60:43103 was elected leader, sending a full tablet report...
I20260926 01:10:24.921521 7839 tablet_bootstrap.cc:492] T 2a5a61fa10f74255b736ba2ccacd5d54 P 2264518cb0fe444da3e1606269628249: No bootstrap required, opened a new log
I20260926 01:10:24.921573 7839 ts_tablet_manager.cc:1397] T 2a5a61fa10f74255b736ba2ccacd5d54 P 2264518cb0fe444da3e1606269628249: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 01:10:24.921595 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:24.921483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45453 (local address 127.0.98.3:41865)
0926 01:10:24.921550 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:24.921572 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260926 01:10:24.921712 7839 raft_consensus.cc:348] T 2a5a61fa10f74255b736ba2ccacd5d54 P 2264518cb0fe444da3e1606269628249 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ffe4fc932934431b944d78dfe891007" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39297 } } peers { permanent_uuid: "72a12d78947e445c85a8d6222cc136af" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43231 } } peers { permanent_uuid: "2264518cb0fe444da3e1606269628249" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35643 } }
I20260926 01:10:24.921772 7839 raft_consensus.cc:374] T 2a5a61fa10f74255b736ba2ccacd5d54 P 2264518cb0fe444da3e1606269628249 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:10:24.921789 7839 raft_consensus.cc:729] T 2a5a61fa10f74255b736ba2ccacd5d54 P 2264518cb0fe444da3e1606269628249 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2264518cb0fe444da3e1606269628249, State: Initialized, Role: FOLLOWER
I20260926 01:10:24.921866 7839 consensus_queue.cc:260] T 2a5a61fa10f74255b736ba2ccacd5d54 P 2264518cb0fe444da3e1606269628249 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ffe4fc932934431b944d78dfe891007" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39297 } } peers { permanent_uuid: "72a12d78947e445c85a8d6222cc136af" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43231 } } peers { permanent_uuid: "2264518cb0fe444da3e1606269628249" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35643 } }
I20260926 01:10:24.922048 7839 ts_tablet_manager.cc:1428] T 2a5a61fa10f74255b736ba2ccacd5d54 P 2264518cb0fe444da3e1606269628249: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:24.922281 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:24.922183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45453 (local address 127.0.98.1:55077)
0926 01:10:24.922228 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:24.922256 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260926 01:10:24.922580 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:24.922465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45453 (local address 127.0.98.1:59283)
0926 01:10:24.922529 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:24.922553 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260926 01:10:24.922631 7460 heartbeater.cc:503] Master 127.0.98.60:43103 was elected leader, sending a full tablet report...
I20260926 01:10:25.056418 7840 raft_consensus.cc:482] T 2a5a61fa10f74255b736ba2ccacd5d54 P 72a12d78947e445c85a8d6222cc136af [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:10:25.056533 7840 raft_consensus.cc:504] T 2a5a61fa10f74255b736ba2ccacd5d54 P 72a12d78947e445c85a8d6222cc136af [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ffe4fc932934431b944d78dfe891007" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39297 } } peers { permanent_uuid: "72a12d78947e445c85a8d6222cc136af" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43231 } } peers { permanent_uuid: "2264518cb0fe444da3e1606269628249" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35643 } }
I20260926 01:10:25.056696 7840 leader_election.cc:288] T 2a5a61fa10f74255b736ba2ccacd5d54 P 72a12d78947e445c85a8d6222cc136af [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1ffe4fc932934431b944d78dfe891007 (127.0.98.2:39297), 2264518cb0fe444da3e1606269628249 (127.0.98.1:35643)
I20260926 01:10:25.056939 7551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a5a61fa10f74255b736ba2ccacd5d54" candidate_uuid: "72a12d78947e445c85a8d6222cc136af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ffe4fc932934431b944d78dfe891007" is_pre_election: true
I20260926 01:10:25.056943 7414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a5a61fa10f74255b736ba2ccacd5d54" candidate_uuid: "72a12d78947e445c85a8d6222cc136af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2264518cb0fe444da3e1606269628249" is_pre_election: true
I20260926 01:10:25.057020 7551 raft_consensus.cc:2454] T 2a5a61fa10f74255b736ba2ccacd5d54 P 1ffe4fc932934431b944d78dfe891007 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 72a12d78947e445c85a8d6222cc136af in term 0.
I20260926 01:10:25.057035 7414 raft_consensus.cc:2454] T 2a5a61fa10f74255b736ba2ccacd5d54 P 2264518cb0fe444da3e1606269628249 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 72a12d78947e445c85a8d6222cc136af in term 0.
I20260926 01:10:25.057190 7622 leader_election.cc:302] T 2a5a61fa10f74255b736ba2ccacd5d54 P 72a12d78947e445c85a8d6222cc136af [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ffe4fc932934431b944d78dfe891007, 72a12d78947e445c85a8d6222cc136af; no voters:
I20260926 01:10:25.057283 7840 raft_consensus.cc:2781] T 2a5a61fa10f74255b736ba2ccacd5d54 P 72a12d78947e445c85a8d6222cc136af [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:10:25.057312 7840 raft_consensus.cc:482] T 2a5a61fa10f74255b736ba2ccacd5d54 P 72a12d78947e445c85a8d6222cc136af [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:10:25.057330 7840 raft_consensus.cc:3037] T 2a5a61fa10f74255b736ba2ccacd5d54 P 72a12d78947e445c85a8d6222cc136af [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:10:25.058064 7840 raft_consensus.cc:504] T 2a5a61fa10f74255b736ba2ccacd5d54 P 72a12d78947e445c85a8d6222cc136af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ffe4fc932934431b944d78dfe891007" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39297 } } peers { permanent_uuid: "72a12d78947e445c85a8d6222cc136af" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43231 } } peers { permanent_uuid: "2264518cb0fe444da3e1606269628249" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35643 } }
I20260926 01:10:25.058250 7840 leader_election.cc:288] T 2a5a61fa10f74255b736ba2ccacd5d54 P 72a12d78947e445c85a8d6222cc136af [CANDIDATE]: Term 1 election: Requested vote from peers 1ffe4fc932934431b944d78dfe891007 (127.0.98.2:39297), 2264518cb0fe444da3e1606269628249 (127.0.98.1:35643)
I20260926 01:10:25.058333 7414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a5a61fa10f74255b736ba2ccacd5d54" candidate_uuid: "72a12d78947e445c85a8d6222cc136af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2264518cb0fe444da3e1606269628249"
I20260926 01:10:25.058405 7414 raft_consensus.cc:3037] T 2a5a61fa10f74255b736ba2ccacd5d54 P 2264518cb0fe444da3e1606269628249 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:10:25.058393 7551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a5a61fa10f74255b736ba2ccacd5d54" candidate_uuid: "72a12d78947e445c85a8d6222cc136af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ffe4fc932934431b944d78dfe891007"
I20260926 01:10:25.058475 7551 raft_consensus.cc:3037] T 2a5a61fa10f74255b736ba2ccacd5d54 P 1ffe4fc932934431b944d78dfe891007 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:10:25.059036 7414 raft_consensus.cc:2454] T 2a5a61fa10f74255b736ba2ccacd5d54 P 2264518cb0fe444da3e1606269628249 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72a12d78947e445c85a8d6222cc136af in term 1.
I20260926 01:10:25.059046 7551 raft_consensus.cc:2454] T 2a5a61fa10f74255b736ba2ccacd5d54 P 1ffe4fc932934431b944d78dfe891007 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72a12d78947e445c85a8d6222cc136af in term 1.
I20260926 01:10:25.059213 7622 leader_election.cc:302] T 2a5a61fa10f74255b736ba2ccacd5d54 P 72a12d78947e445c85a8d6222cc136af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ffe4fc932934431b944d78dfe891007, 72a12d78947e445c85a8d6222cc136af; no voters:
I20260926 01:10:25.059315 7840 raft_consensus.cc:2781] T 2a5a61fa10f74255b736ba2ccacd5d54 P 72a12d78947e445c85a8d6222cc136af [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:10:25.059463 7840 raft_consensus.cc:686] T 2a5a61fa10f74255b736ba2ccacd5d54 P 72a12d78947e445c85a8d6222cc136af [term 1 LEADER]: Becoming Leader. State: Replica: 72a12d78947e445c85a8d6222cc136af, State: Running, Role: LEADER
I20260926 01:10:25.059507 7840 consensus_queue.cc:237] T 2a5a61fa10f74255b736ba2ccacd5d54 P 72a12d78947e445c85a8d6222cc136af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ffe4fc932934431b944d78dfe891007" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39297 } } peers { permanent_uuid: "72a12d78947e445c85a8d6222cc136af" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43231 } } peers { permanent_uuid: "2264518cb0fe444da3e1606269628249" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35643 } }
I20260926 01:10:25.059823 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:25.059724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45453 (local address 127.0.98.3:54251)
0926 01:10:25.059761 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:25.059794 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260926 01:10:25.060204 7280 catalog_manager.cc:5495] T 2a5a61fa10f74255b736ba2ccacd5d54 P 72a12d78947e445c85a8d6222cc136af reported cstate change: term changed from 0 to 1, leader changed from <none> to 72a12d78947e445c85a8d6222cc136af (127.0.98.3). New cstate: current_term: 1 leader_uuid: "72a12d78947e445c85a8d6222cc136af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ffe4fc932934431b944d78dfe891007" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39297 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "72a12d78947e445c85a8d6222cc136af" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2264518cb0fe444da3e1606269628249" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35643 } health_report { overall_health: UNKNOWN } } }
I20260926 01:10:25.097468 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:25.095381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53190 (local address 127.0.98.3:43231)
0926 01:10:25.095429 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:25.095431 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:25.095441 (+ 10us) server_negotiation.cc:407] Connection header received
0926 01:10:25.095472 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:25.095473 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:25.095485 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:25.095533 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:10:25.095696 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:25.096201 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:25.096964 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:25.097151 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:25.097202 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:25.097293 (+ 91us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:10:25.097319 (+ 26us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:10:25.097352 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:25.097391 (+ 39us) server_negotiation.cc:299] Negotiation successful
0926 01:10:25.097409 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 01:10:25.098343 7414 raft_consensus.cc:1261] T 2a5a61fa10f74255b736ba2ccacd5d54 P 2264518cb0fe444da3e1606269628249 [term 1 FOLLOWER]: Refusing update from remote peer 72a12d78947e445c85a8d6222cc136af: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:10:25.098369 7551 raft_consensus.cc:1261] T 2a5a61fa10f74255b736ba2ccacd5d54 P 1ffe4fc932934431b944d78dfe891007 [term 1 FOLLOWER]: Refusing update from remote peer 72a12d78947e445c85a8d6222cc136af: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:10:25.098578 7840 consensus_queue.cc:1035] T 2a5a61fa10f74255b736ba2ccacd5d54 P 72a12d78947e445c85a8d6222cc136af [LEADER]: Connected to new peer: Peer: permanent_uuid: "2264518cb0fe444da3e1606269628249" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:10:25.098696 7843 consensus_queue.cc:1035] T 2a5a61fa10f74255b736ba2ccacd5d54 P 72a12d78947e445c85a8d6222cc136af [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ffe4fc932934431b944d78dfe891007" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:10:25.099010 7597 heartbeater.cc:503] Master 127.0.98.60:43103 was elected leader, sending a full tablet report...
I20260926 01:10:25.099467 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:25.099246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45453 (local address 127.0.98.2:47515)
0926 01:10:25.099396 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:25.099435 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131}
I20260926 01:10:25.099601 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:25.099325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45453 (local address 127.0.98.1:44881)
0926 01:10:25.099545 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:25.099574 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204}
I20260926 01:10:25.100351 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:25.100258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45453 (local address 127.0.98.3:54147)
0926 01:10:25.100290 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:25.100322 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260926 01:10:25.101127 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:25.101033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45453 (local address 127.0.98.1:37919)
0926 01:10:25.101068 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:25.101100 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260926 01:10:25.101382 7846 mvcc.cc:205] Tried to move back new op lower bound from 7333417062801170432 to 7333417062644371456. Current Snapshot: MvccSnapshot[applied={T|T < 7333417062801170432 or (T in {7333417062801170432})}]
I20260926 01:10:25.101641 7280 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49128:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:10:25.101730 7280 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49128:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:10:25.102316 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:25.102227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45453 (local address 127.0.98.2:54165)
0926 01:10:25.102260 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:25.102287 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260926 01:10:25.101745 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:25.101664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45453 (local address 127.0.98.3:46493)
0926 01:10:25.101696 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:25.101720 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260926 01:10:25.103173 7280 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3b48fee3b6474a23a70b62be3d2b7f2f: Sending DeleteTablet for 3 replicas of tablet 2a5a61fa10f74255b736ba2ccacd5d54
I20260926 01:10:25.103374 7531 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a5a61fa10f74255b736ba2ccacd5d54 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:10:25 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:41024
I20260926 01:10:25.103468 7667 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a5a61fa10f74255b736ba2ccacd5d54 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:10:25 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:53188
I20260926 01:10:25.103493 7394 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a5a61fa10f74255b736ba2ccacd5d54 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:10:25 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:60358
I20260926 01:10:25.103565 7855 tablet_replica.cc:330] T 2a5a61fa10f74255b736ba2ccacd5d54 P 1ffe4fc932934431b944d78dfe891007: stopping tablet replica
I20260926 01:10:25.103614 7855 raft_consensus.cc:2229] T 2a5a61fa10f74255b736ba2ccacd5d54 P 1ffe4fc932934431b944d78dfe891007 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:10:25.103641 7856 tablet_replica.cc:330] T 2a5a61fa10f74255b736ba2ccacd5d54 P 72a12d78947e445c85a8d6222cc136af: stopping tablet replica
I20260926 01:10:25.103645 7857 tablet_replica.cc:330] T 2a5a61fa10f74255b736ba2ccacd5d54 P 2264518cb0fe444da3e1606269628249: stopping tablet replica
I20260926 01:10:25.103649 7855 raft_consensus.cc:2258] T 2a5a61fa10f74255b736ba2ccacd5d54 P 1ffe4fc932934431b944d78dfe891007 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:10:25.103682 7856 raft_consensus.cc:2229] T 2a5a61fa10f74255b736ba2ccacd5d54 P 72a12d78947e445c85a8d6222cc136af [term 1 LEADER]: Raft consensus shutting down.
I20260926 01:10:25.103682 7857 raft_consensus.cc:2229] T 2a5a61fa10f74255b736ba2ccacd5d54 P 2264518cb0fe444da3e1606269628249 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:10:25.103732 7857 raft_consensus.cc:2258] T 2a5a61fa10f74255b736ba2ccacd5d54 P 2264518cb0fe444da3e1606269628249 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:10:25.103745 7856 raft_consensus.cc:2258] T 2a5a61fa10f74255b736ba2ccacd5d54 P 72a12d78947e445c85a8d6222cc136af [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:10:25.104010 7857 ts_tablet_manager.cc:1905] T 2a5a61fa10f74255b736ba2ccacd5d54 P 2264518cb0fe444da3e1606269628249: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:10:25.104010 7855 ts_tablet_manager.cc:1905] T 2a5a61fa10f74255b736ba2ccacd5d54 P 1ffe4fc932934431b944d78dfe891007: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:10:25.104035 7531 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a5a61fa10f74255b736ba2ccacd5d54 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:10:25 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:41024
I20260926 01:10:25.104403 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 7235
I20260926 01:10:25.104611 7856 ts_tablet_manager.cc:1905] T 2a5a61fa10f74255b736ba2ccacd5d54 P 72a12d78947e445c85a8d6222cc136af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:10:25.106805 7855 ts_tablet_manager.cc:1918] T 2a5a61fa10f74255b736ba2ccacd5d54 P 1ffe4fc932934431b944d78dfe891007: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:10:25.106863 7855 log.cc:1196] T 2a5a61fa10f74255b736ba2ccacd5d54 P 1ffe4fc932934431b944d78dfe891007: Deleting WAL directory at /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/ts-1/wal/wals/2a5a61fa10f74255b736ba2ccacd5d54
I20260926 01:10:25.107090 7855 ts_tablet_manager.cc:1939] T 2a5a61fa10f74255b736ba2ccacd5d54 P 1ffe4fc932934431b944d78dfe891007: Deleting consensus metadata
I20260926 01:10:25.107409 7267 catalog_manager.cc:4841] TS 1ffe4fc932934431b944d78dfe891007 (127.0.98.2:39297): tablet 2a5a61fa10f74255b736ba2ccacd5d54 (table test-table [id=2efcbd4e061449c7b32691ce889a8ef1]) successfully deleted
I20260926 01:10:25.107565 7856 ts_tablet_manager.cc:1918] T 2a5a61fa10f74255b736ba2ccacd5d54 P 72a12d78947e445c85a8d6222cc136af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260926 01:10:25.107573 7267 catalog_manager.cc:4808] TS 1ffe4fc932934431b944d78dfe891007 (127.0.98.2:39297): delete failed for tablet 2a5a61fa10f74255b736ba2ccacd5d54 because the tablet was not found. No further retry: Not found: Tablet not found: 2a5a61fa10f74255b736ba2ccacd5d54
I20260926 01:10:25.107622 7856 log.cc:1196] T 2a5a61fa10f74255b736ba2ccacd5d54 P 72a12d78947e445c85a8d6222cc136af: Deleting WAL directory at /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384993574762-392-0/minicluster-data/ts-2/wal/wals/2a5a61fa10f74255b736ba2ccacd5d54
I20260926 01:10:25.107822 7856 ts_tablet_manager.cc:1939] T 2a5a61fa10f74255b736ba2ccacd5d54 P 72a12d78947e445c85a8d6222cc136af: Deleting consensus metadata
I20260926 01:10:25.108271 7265 catalog_manager.cc:4841] TS 72a12d78947e445c85a8d6222cc136af (127.0.98.3:43231): tablet 2a5a61fa10f74255b736ba2ccacd5d54 (table test-table [id=2efcbd4e061449c7b32691ce889a8ef1]) successfully deleted
I20260926 01:10:25.109694 7831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:25.109601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35643 (local address 127.0.0.1:60366)
0926 01:10:25.109637 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:25.109661 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20260926 01:10:25.109874 7266 catalog_manager.cc:4584] TS 2264518cb0fe444da3e1606269628249 (127.0.98.1:35643): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2a5a61fa10f74255b736ba2ccacd5d54: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35643: connect: Connection refused (error 111)
I20260926 01:10:25.109911 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 7468
I20260926 01:10:25.114524 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 7604
I20260926 01:10:25.119330 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 6955
I20260926 01:10:25.123250 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 7142
2026-09-26T01:10:25Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (6559 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384993574762-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 01:10:25 dist-test-slave-7b22 krb5kdc[7860](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 01:10:25 dist-test-slave-7b22 krb5kdc[7860](info): set up 2 sockets
Sep 26 01:10:25 dist-test-slave-7b22 krb5kdc[7860](info): commencing operation
krb5kdc: starting...
W20260926 01:10:26.081979 392 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384993574762-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384993574762-392-0/krb5kdc/test-user.keytab.
Sep 26 01:10:26 dist-test-slave-7b22 krb5kdc[7860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385026, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T01:10:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:10:26Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384993574762-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384993574762-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260926 01:10:26.197178 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384993574762-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384993574762-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384993574762-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35021
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384993574762-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35021
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:10:26.276953 7873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:10:26.277168 7873 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:10:26.277199 7873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:10:26.278417 7873 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:10:26.278458 7873 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 01:10:26.278474 7873 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:10:26.278491 7873 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:10:26.278507 7873 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:10:26.278522 7873 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:10:26.279974 7873 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384993574762-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384993574762-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35021
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384993574762-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35021
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7873
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384993574762-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:10:26.280211 7873 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:10:26.280416 7873 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 01:10:26.285145 7873 server_base.cc:1034] running on GCE node
W20260926 01:10:26.285157 7878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:10:26.285120 7881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:10:26.285126 7879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:10:26.285542 7873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:10:26.285957 7873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:10:26.287151 7873 hybrid_clock.cc:648] HybridClock initialized: now 1790385026287134 us; error 32 us; skew 500 ppm
Sep 26 01:10:26 dist-test-slave-7b22 krb5kdc[7860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385026, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:10:26.295039 7873 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260926 01:10:26.295585 7873 webserver.cc:460] Webserver started at http://127.0.98.62:33141/ using document root <none> and password file <none>
I20260926 01:10:26.295826 7873 fs_manager.cc:359] Metadata directory not provided
I20260926 01:10:26.295883 7873 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:10:26.296022 7873 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:10:26.297120 7873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384993574762-392-0/minicluster-data/master-0/data/instance:
uuid: "142a74e6326f4805a91f83838eeda07e"
format_stamp: "Formatted at 2026-09-26 01:10:26 on dist-test-slave-7b22"
server_key: "4aea3dc70f4ea22d13c9d7159daebe6b"
server_key_iv: "af5a56c7d8258672859a270b700d1d7d"
server_key_version: "encryptionkey@0"
I20260926 01:10:26.297477 7873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384993574762-392-0/minicluster-data/master-0/wal/instance:
uuid: "142a74e6326f4805a91f83838eeda07e"
format_stamp: "Formatted at 2026-09-26 01:10:26 on dist-test-slave-7b22"
server_key: "4aea3dc70f4ea22d13c9d7159daebe6b"
server_key_iv: "af5a56c7d8258672859a270b700d1d7d"
server_key_version: "encryptionkey@0"
I20260926 01:10:26.298930 7873 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 01:10:26.299805 7888 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:26.300014 7873 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 01:10:26.300107 7873 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384993574762-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384993574762-392-0/minicluster-data/master-0/wal
uuid: "142a74e6326f4805a91f83838eeda07e"
format_stamp: "Formatted at 2026-09-26 01:10:26 on dist-test-slave-7b22"
server_key: "4aea3dc70f4ea22d13c9d7159daebe6b"
server_key_iv: "af5a56c7d8258672859a270b700d1d7d"
server_key_version: "encryptionkey@0"
I20260926 01:10:26.300172 7873 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384993574762-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384993574762-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384993574762-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:10:26.307294 7873 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:10:26.309237 7873 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:10:26.309368 7873 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:10:26.313656 7873 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35021
I20260926 01:10:26.313694 7950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35021 every 8 connection(s)
I20260926 01:10:26.314047 7873 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
I20260926 01:10:26.314661 7951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:10:26.316811 7951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 142a74e6326f4805a91f83838eeda07e: Bootstrap starting.
I20260926 01:10:26.317318 7951 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 142a74e6326f4805a91f83838eeda07e: Neither blocks nor log segments found. Creating new log.
I20260926 01:10:26.317548 7951 log.cc:824] T 00000000000000000000000000000000 P 142a74e6326f4805a91f83838eeda07e: Log is configured to *not* fsync() on all Append() calls
I20260926 01:10:26.318185 7951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 142a74e6326f4805a91f83838eeda07e: No bootstrap required, opened a new log
I20260926 01:10:26.320185 7951 raft_consensus.cc:348] T 00000000000000000000000000000000 P 142a74e6326f4805a91f83838eeda07e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "142a74e6326f4805a91f83838eeda07e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35021 } }
I20260926 01:10:26.320328 7951 raft_consensus.cc:374] T 00000000000000000000000000000000 P 142a74e6326f4805a91f83838eeda07e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:10:26.320349 7951 raft_consensus.cc:729] T 00000000000000000000000000000000 P 142a74e6326f4805a91f83838eeda07e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 142a74e6326f4805a91f83838eeda07e, State: Initialized, Role: FOLLOWER
I20260926 01:10:26.320446 7951 consensus_queue.cc:260] T 00000000000000000000000000000000 P 142a74e6326f4805a91f83838eeda07e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "142a74e6326f4805a91f83838eeda07e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35021 } }
I20260926 01:10:26.320499 7951 raft_consensus.cc:388] T 00000000000000000000000000000000 P 142a74e6326f4805a91f83838eeda07e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:10:26.320525 7951 raft_consensus.cc:482] T 00000000000000000000000000000000 P 142a74e6326f4805a91f83838eeda07e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:10:26.320561 7951 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 142a74e6326f4805a91f83838eeda07e [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:10:26.321075 7951 raft_consensus.cc:504] T 00000000000000000000000000000000 P 142a74e6326f4805a91f83838eeda07e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "142a74e6326f4805a91f83838eeda07e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35021 } }
I20260926 01:10:26.321166 7951 leader_election.cc:302] T 00000000000000000000000000000000 P 142a74e6326f4805a91f83838eeda07e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 142a74e6326f4805a91f83838eeda07e; no voters:
I20260926 01:10:26.321316 7951 leader_election.cc:288] T 00000000000000000000000000000000 P 142a74e6326f4805a91f83838eeda07e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:10:26.321410 7954 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 142a74e6326f4805a91f83838eeda07e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:10:26.321560 7954 raft_consensus.cc:686] T 00000000000000000000000000000000 P 142a74e6326f4805a91f83838eeda07e [term 1 LEADER]: Becoming Leader. State: Replica: 142a74e6326f4805a91f83838eeda07e, State: Running, Role: LEADER
I20260926 01:10:26.321585 7951 sys_catalog.cc:564] T 00000000000000000000000000000000 P 142a74e6326f4805a91f83838eeda07e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:10:26.321652 7954 consensus_queue.cc:237] T 00000000000000000000000000000000 P 142a74e6326f4805a91f83838eeda07e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "142a74e6326f4805a91f83838eeda07e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35021 } }
I20260926 01:10:26.321805 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 7873
I20260926 01:10:26.321941 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384993574762-392-0/minicluster-data/master-0/wal/instance
I20260926 01:10:26.322136 392 external_mini_cluster.cc:1428] Setting key 60c017ed2564880739e3fd3fb7849441
I20260926 01:10:26.322196 7955 sys_catalog.cc:455] T 00000000000000000000000000000000 P 142a74e6326f4805a91f83838eeda07e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "142a74e6326f4805a91f83838eeda07e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "142a74e6326f4805a91f83838eeda07e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35021 } } }
I20260926 01:10:26.322258 7955 sys_catalog.cc:458] T 00000000000000000000000000000000 P 142a74e6326f4805a91f83838eeda07e [sys.catalog]: This master's current role is: LEADER
I20260926 01:10:26.322539 7956 sys_catalog.cc:455] T 00000000000000000000000000000000 P 142a74e6326f4805a91f83838eeda07e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 142a74e6326f4805a91f83838eeda07e. Latest consensus state: current_term: 1 leader_uuid: "142a74e6326f4805a91f83838eeda07e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "142a74e6326f4805a91f83838eeda07e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35021 } } }
I20260926 01:10:26.322613 7956 sys_catalog.cc:458] T 00000000000000000000000000000000 P 142a74e6326f4805a91f83838eeda07e [sys.catalog]: This master's current role is: LEADER
I20260926 01:10:26.322948 7963 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:10:26.323444 7963 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:10:26.324905 7963 catalog_manager.cc:1347] Generated new cluster ID: 107c6d7559a4440fa22a1b565b1c2cc9
I20260926 01:10:26.324952 7963 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 01:10:26 dist-test-slave-7b22 krb5kdc[7860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385026, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 01:10:26.330845 7963 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:10:26.331586 7963 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:10:26.332605 7967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:26.322606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47780 (local address 127.0.98.62:35021)
0926 01:10:26.322760 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:26.322770 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:26.322835 (+ 65us) server_negotiation.cc:407] Connection header received
0926 01:10:26.322963 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:26.322971 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:26.323032 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:26.323174 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:26.323910 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:26.325051 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:26.326116 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:26.326700 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:26.326794 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:26.329469 (+ 2675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:10:26.329474 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:10:26.329474 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:10:26.329487 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:10:26.331266 (+ 1779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:26.331569 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:26.331572 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:26.331573 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:26.331651 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:26.331817 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:26.331818 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:26.331818 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:26.332025 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:10:26.332168 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:26.332259 (+ 91us) server_negotiation.cc:299] Negotiation successful
0926 01:10:26.332428 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":60,"threads_started":1}
I20260926 01:10:26.334700 7963 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 142a74e6326f4805a91f83838eeda07e: Generated new TSK 0
I20260926 01:10:26.334851 7963 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384993574762-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384993574762-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260926 01:10:26.363337 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384993574762-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384993574762-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384993574762-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384993574762-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35021
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384993574762-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:10:26.444312 7977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:10:26.444517 7977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:10:26.444550 7977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:10:26.445744 7977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:10:26.445787 7977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:10:26.445842 7977 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 01:10:26.445873 7977 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 01:10:26.447392 7977 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384993574762-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384993574762-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384993574762-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384993574762-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:35021
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7977
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384993574762-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:10:26.447611 7977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:10:26.447827 7977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 01:10:26.452584 7977 server_base.cc:1034] running on GCE node
W20260926 01:10:26.452521 7982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:10:26.452512 7983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:10:26.452518 7985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:10:26.452973 7977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:10:26.453171 7977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:10:26.454294 7977 hybrid_clock.cc:648] HybridClock initialized: now 1790385026454285 us; error 26 us; skew 500 ppm
Sep 26 01:10:26 dist-test-slave-7b22 krb5kdc[7860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385026, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:10:26.456640 7977 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260926 01:10:26.457026 7977 webserver.cc:460] Webserver started at http://127.0.98.1:42637/ using document root <none> and password file <none>
I20260926 01:10:26.457207 7977 fs_manager.cc:359] Metadata directory not provided
I20260926 01:10:26.457255 7977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:10:26.457347 7977 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:10:26.458391 7977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384993574762-392-0/minicluster-data/ts-0/data/instance:
uuid: "e69c8d1efa23415a8c293981f63d9449"
format_stamp: "Formatted at 2026-09-26 01:10:26 on dist-test-slave-7b22"
server_key: "64eb75f1321fa290705103c448106325"
server_key_iv: "eb890494196d4bd79c90be8eed917784"
server_key_version: "encryptionkey@0"
I20260926 01:10:26.458676 7977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384993574762-392-0/minicluster-data/ts-0/wal/instance:
uuid: "e69c8d1efa23415a8c293981f63d9449"
format_stamp: "Formatted at 2026-09-26 01:10:26 on dist-test-slave-7b22"
server_key: "64eb75f1321fa290705103c448106325"
server_key_iv: "eb890494196d4bd79c90be8eed917784"
server_key_version: "encryptionkey@0"
I20260926 01:10:26.459800 7977 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 01:10:26.460402 7992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:26.460563 7977 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 01:10:26.460635 7977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384993574762-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384993574762-392-0/minicluster-data/ts-0/wal
uuid: "e69c8d1efa23415a8c293981f63d9449"
format_stamp: "Formatted at 2026-09-26 01:10:26 on dist-test-slave-7b22"
server_key: "64eb75f1321fa290705103c448106325"
server_key_iv: "eb890494196d4bd79c90be8eed917784"
server_key_version: "encryptionkey@0"
I20260926 01:10:26.460688 7977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384993574762-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384993574762-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384993574762-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:10:26.483235 7977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:10:26.485421 7977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:10:26.485538 7977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:10:26.486076 7977 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:10:26.486117 7977 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:26.486140 7977 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:10:26.486151 7977 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:26.492342 7977 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46585
I20260926 01:10:26.492488 8107 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46585 every 8 connection(s)
Sep 26 01:10:26 dist-test-slave-7b22 krb5kdc[7860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385026, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 01:10:26.492771 7977 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384993574762-392-0/minicluster-data/ts-0/data/info.pb
I20260926 01:10:26.495949 7967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:26.486513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45597 (local address 127.0.98.62:35021)
0926 01:10:26.486578 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:26.486581 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:26.486736 (+ 155us) server_negotiation.cc:407] Connection header received
0926 01:10:26.487293 (+ 557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:26.487294 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:26.487313 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:26.487395 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:26.488186 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:26.488982 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:26.490548 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:26.490832 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:26.490905 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:26.493244 (+ 2339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:10:26.493252 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:10:26.493253 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:10:26.493259 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:10:26.494548 (+ 1289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:26.494926 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:26.494927 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:26.494927 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:26.494989 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:26.495252 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:26.495253 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:26.495254 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:26.495393 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:10:26.495457 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:26.495744 (+ 287us) server_negotiation.cc:299] Negotiation successful
0926 01:10:26.495849 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 01:10:26.496109 8001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:26.486472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35021 (local address 127.0.98.1:45597)
0926 01:10:26.486614 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:26.486654 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:26.487193 (+ 539us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:26.487458 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:26.487461 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:26.487786 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:10:26.488064 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:26.488070 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:26.489127 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:26.489129 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:26.490463 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:26.490466 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:26.490921 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:26.490921 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:26.491020 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:26.491560 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:10:26.491570 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:10:26.493166 (+ 1596us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:10:26.494614 (+ 1448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:10:26.494618 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:10:26.494620 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:10:26.494844 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:10:26.495055 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:10:26.495056 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:10:26.495057 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:10:26.495185 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:10:26.495458 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:10:26.495462 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:10:26.495613 (+ 151us) client_negotiation.cc:769] Sending connection context
0926 01:10:26.495714 (+ 101us) client_negotiation.cc:240] Negotiation successful
0926 01:10:26.495880 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":76,"threads_started":1}
I20260926 01:10:26.496747 8108 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35021
I20260926 01:10:26.496872 8108 heartbeater.cc:464] Registering TS with master...
I20260926 01:10:26.497094 8108 heartbeater.cc:511] Master 127.0.98.62:35021 requested a full tablet report, sending...
I20260926 01:10:26.497795 7904 ts_manager.cc:194] Registered new tserver with Master: e69c8d1efa23415a8c293981f63d9449 (127.0.98.1:46585)
I20260926 01:10:26.498714 7904 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:45597
I20260926 01:10:26.499667 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 7977
I20260926 01:10:26.499788 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384993574762-392-0/minicluster-data/ts-0/wal/instance
I20260926 01:10:26.500003 392 external_mini_cluster.cc:1428] Setting key 4ec15fdb183588ba5a7b29ee623a490f
I20260926 01:10:26.503819 7967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:26.497772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38477 (local address 127.0.98.62:35021)
0926 01:10:26.497850 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:26.497852 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:26.498037 (+ 185us) server_negotiation.cc:407] Connection header received
0926 01:10:26.498108 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:26.498109 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:26.498119 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:26.498183 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:26.498511 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:26.499147 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:26.500139 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:26.500407 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:26.500474 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:26.501626 (+ 1152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:10:26.501628 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:10:26.501628 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:10:26.501634 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:10:26.502885 (+ 1251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:26.503217 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:26.503218 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:26.503218 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:26.503280 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:26.503442 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:26.503443 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:26.503443 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:26.503526 (+ 83us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:10:26.503568 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:26.503707 (+ 139us) server_negotiation.cc:299] Negotiation successful
0926 01:10:26.503761 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 01:10:26.503885 8114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:26.497787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35021 (local address 127.0.98.1:38477)
0926 01:10:26.497972 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:26.497981 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:26.498049 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:26.498164 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:26.498165 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:26.498360 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:10:26.498424 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:26.498426 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:26.499258 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:26.499259 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:26.500052 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:26.500054 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:26.500473 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:26.500474 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:26.500557 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:26.501057 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:10:26.501062 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:10:26.501524 (+ 462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:10:26.502954 (+ 1430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:10:26.502955 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:10:26.502955 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:10:26.503155 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:10:26.503331 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:10:26.503331 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:10:26.503331 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:10:26.503391 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:10:26.503573 (+ 182us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:10:26.503574 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:10:26.503618 (+ 44us) client_negotiation.cc:769] Sending connection context
0926 01:10:26.503692 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 01:10:26.503781 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"mutex_wait_us":2,"thread_start_us":94,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384993574762-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384993574762-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260926 01:10:26.525540 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384993574762-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384993574762-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384993574762-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384993574762-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35021
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384993574762-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:10:26.604501 8117 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:10:26.604702 8117 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:10:26.604727 8117 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:10:26.605872 8117 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:10:26.605906 8117 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:10:26.605924 8117 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 01:10:26.605940 8117 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 01:10:26.607368 8117 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384993574762-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384993574762-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384993574762-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384993574762-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:35021
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8117
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384993574762-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:10:26.607559 8117 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:10:26.607738 8117 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:10:26.612641 8123 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:10:26.612668 8117 server_base.cc:1034] running on GCE node
W20260926 01:10:26.612690 8125 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:10:26.612671 8122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:10:26.613065 8117 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:10:26.613266 8117 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:10:26.614409 8117 hybrid_clock.cc:648] HybridClock initialized: now 1790385026614396 us; error 32 us; skew 500 ppm
Sep 26 01:10:26 dist-test-slave-7b22 krb5kdc[7860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385026, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:10:26.616911 8117 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260926 01:10:26.617332 8117 webserver.cc:460] Webserver started at http://127.0.98.2:33987/ using document root <none> and password file <none>
I20260926 01:10:26.617537 8117 fs_manager.cc:359] Metadata directory not provided
I20260926 01:10:26.617583 8117 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:10:26.617705 8117 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:10:26.618788 8117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384993574762-392-0/minicluster-data/ts-1/data/instance:
uuid: "feef267fe1a44b048ebeb322ca16140b"
format_stamp: "Formatted at 2026-09-26 01:10:26 on dist-test-slave-7b22"
server_key: "463b746eb17cdbf0f19e35af615a2be4"
server_key_iv: "6696d229cabc009853dcb137dc4d6a9a"
server_key_version: "encryptionkey@0"
I20260926 01:10:26.619104 8117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384993574762-392-0/minicluster-data/ts-1/wal/instance:
uuid: "feef267fe1a44b048ebeb322ca16140b"
format_stamp: "Formatted at 2026-09-26 01:10:26 on dist-test-slave-7b22"
server_key: "463b746eb17cdbf0f19e35af615a2be4"
server_key_iv: "6696d229cabc009853dcb137dc4d6a9a"
server_key_version: "encryptionkey@0"
I20260926 01:10:26.620399 8117 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 01:10:26.621227 8132 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:26.621444 8117 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:10:26.621536 8117 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384993574762-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384993574762-392-0/minicluster-data/ts-1/wal
uuid: "feef267fe1a44b048ebeb322ca16140b"
format_stamp: "Formatted at 2026-09-26 01:10:26 on dist-test-slave-7b22"
server_key: "463b746eb17cdbf0f19e35af615a2be4"
server_key_iv: "6696d229cabc009853dcb137dc4d6a9a"
server_key_version: "encryptionkey@0"
I20260926 01:10:26.621606 8117 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384993574762-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384993574762-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384993574762-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:10:26.632654 8117 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:10:26.634835 8117 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:10:26.634986 8117 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:10:26.635488 8117 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:10:26.635531 8117 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:26.635563 8117 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:10:26.635581 8117 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:26.642526 8117 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44745
I20260926 01:10:26.642666 8247 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44745 every 8 connection(s)
Sep 26 01:10:26 dist-test-slave-7b22 krb5kdc[7860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385026, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 01:10:26.642951 8117 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384993574762-392-0/minicluster-data/ts-1/data/info.pb
I20260926 01:10:26.645798 7967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:26.635966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41379 (local address 127.0.98.62:35021)
0926 01:10:26.636012 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:26.636013 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:26.636231 (+ 218us) server_negotiation.cc:407] Connection header received
0926 01:10:26.636769 (+ 538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:26.636770 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:26.636783 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:26.636847 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:26.637912 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:26.638676 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:26.640255 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:26.640542 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:26.640616 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:26.643394 (+ 2778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:10:26.643396 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:10:26.643396 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:10:26.643404 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:10:26.644505 (+ 1101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:26.644896 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:26.644896 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:26.644897 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:26.644967 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:26.645225 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:26.645225 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:26.645225 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:26.645329 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:10:26.645377 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:26.645672 (+ 295us) server_negotiation.cc:299] Negotiation successful
0926 01:10:26.645734 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 01:10:26.646107 8141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:26.635970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35021 (local address 127.0.98.2:41379)
0926 01:10:26.636117 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:26.636152 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:26.636667 (+ 515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:26.636969 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:26.636975 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:26.637360 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:10:26.637790 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:26.637799 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:26.638775 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:26.638776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:26.640152 (+ 1376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:26.640156 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:26.640627 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:26.640628 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:26.640820 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:26.641498 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:10:26.641509 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:10:26.643299 (+ 1790us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:10:26.644584 (+ 1285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:10:26.644590 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:10:26.644592 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:10:26.644809 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:10:26.645048 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:10:26.645049 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:10:26.645049 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:10:26.645166 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:10:26.645408 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:10:26.645414 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:10:26.645544 (+ 130us) client_negotiation.cc:769] Sending connection context
0926 01:10:26.645655 (+ 111us) client_negotiation.cc:240] Negotiation successful
0926 01:10:26.645854 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":74,"threads_started":1}
I20260926 01:10:26.646595 8248 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35021
I20260926 01:10:26.646687 8248 heartbeater.cc:464] Registering TS with master...
I20260926 01:10:26.646942 8248 heartbeater.cc:511] Master 127.0.98.62:35021 requested a full tablet report, sending...
I20260926 01:10:26.647356 7905 ts_manager.cc:194] Registered new tserver with Master: feef267fe1a44b048ebeb322ca16140b (127.0.98.2:44745)
I20260926 01:10:26.647859 7905 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:41379
I20260926 01:10:26.650316 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 8117
I20260926 01:10:26.650434 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384993574762-392-0/minicluster-data/ts-1/wal/instance
I20260926 01:10:26.650635 392 external_mini_cluster.cc:1428] Setting key 6c115e449b56f1dadbb41f854b7001ce
I20260926 01:10:26.653205 8254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:26.647652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35021 (local address 127.0.98.2:54729)
0926 01:10:26.647785 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:26.647795 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:26.647845 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:26.647976 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:26.647977 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:26.648176 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:10:26.648241 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:26.648243 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:26.649003 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:26.649003 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:26.649532 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:26.649533 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:26.649903 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:26.649904 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:26.649957 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:26.650481 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:10:26.650486 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:10:26.650838 (+ 352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:10:26.652163 (+ 1325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:10:26.652164 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:10:26.652164 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:10:26.652363 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:10:26.652569 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:10:26.652570 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:10:26.652570 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:10:26.652628 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:10:26.652931 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:10:26.652932 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:10:26.652974 (+ 42us) client_negotiation.cc:769] Sending connection context
0926 01:10:26.653048 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 01:10:26.653130 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":85,"threads_started":1}
I20260926 01:10:26.653249 7967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:26.647735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54729 (local address 127.0.98.62:35021)
0926 01:10:26.647777 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:26.647779 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:26.647839 (+ 60us) server_negotiation.cc:407] Connection header received
0926 01:10:26.647885 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:26.647886 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:26.647897 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:26.647965 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:26.648316 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:26.648917 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:26.649603 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:26.649792 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:26.649904 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:26.650911 (+ 1007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:10:26.650913 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:10:26.650913 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:10:26.650920 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:10:26.652086 (+ 1166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:26.652437 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:26.652438 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:26.652439 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:26.652505 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:26.652675 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:26.652675 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:26.652676 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:26.652849 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:10:26.652912 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:26.653079 (+ 167us) server_negotiation.cc:299] Negotiation successful
0926 01:10:26.653158 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384993574762-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384993574762-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260926 01:10:26.676534 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384993574762-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384993574762-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384993574762-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384993574762-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35021
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384993574762-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:10:26.755857 8257 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:10:26.756073 8257 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:10:26.756098 8257 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:10:26.757234 8257 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:10:26.757272 8257 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:10:26.757290 8257 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 01:10:26.757306 8257 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 01:10:26.758777 8257 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384993574762-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384993574762-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384993574762-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384993574762-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:35021
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8257
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384993574762-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:10:26.758975 8257 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:10:26.759150 8257 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:10:26.764006 8262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:10:26.764196 8257 server_base.cc:1034] running on GCE node
W20260926 01:10:26.764034 8265 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:10:26.764011 8263 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:10:26.764490 8257 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:10:26.764720 8257 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:10:26.765904 8257 hybrid_clock.cc:648] HybridClock initialized: now 1790385026765884 us; error 31 us; skew 500 ppm
Sep 26 01:10:26 dist-test-slave-7b22 krb5kdc[7860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385026, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:10:26.768409 8257 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260926 01:10:26.768877 8257 webserver.cc:460] Webserver started at http://127.0.98.3:41399/ using document root <none> and password file <none>
I20260926 01:10:26.769088 8257 fs_manager.cc:359] Metadata directory not provided
I20260926 01:10:26.769134 8257 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:10:26.769253 8257 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:10:26.770321 8257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384993574762-392-0/minicluster-data/ts-2/data/instance:
uuid: "39a6c0acb49147dcaa253562329fdde4"
format_stamp: "Formatted at 2026-09-26 01:10:26 on dist-test-slave-7b22"
server_key: "a382251958e4c5e9fdc876fd9ea420af"
server_key_iv: "a9580fde76d25a5116555e100c31ad66"
server_key_version: "encryptionkey@0"
I20260926 01:10:26.770639 8257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384993574762-392-0/minicluster-data/ts-2/wal/instance:
uuid: "39a6c0acb49147dcaa253562329fdde4"
format_stamp: "Formatted at 2026-09-26 01:10:26 on dist-test-slave-7b22"
server_key: "a382251958e4c5e9fdc876fd9ea420af"
server_key_iv: "a9580fde76d25a5116555e100c31ad66"
server_key_version: "encryptionkey@0"
I20260926 01:10:26.771888 8257 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 01:10:26.772610 8272 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:26.772768 8257 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 01:10:26.772863 8257 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384993574762-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384993574762-392-0/minicluster-data/ts-2/wal
uuid: "39a6c0acb49147dcaa253562329fdde4"
format_stamp: "Formatted at 2026-09-26 01:10:26 on dist-test-slave-7b22"
server_key: "a382251958e4c5e9fdc876fd9ea420af"
server_key_iv: "a9580fde76d25a5116555e100c31ad66"
server_key_version: "encryptionkey@0"
I20260926 01:10:26.772925 8257 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384993574762-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384993574762-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384993574762-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:10:26.807859 8257 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:10:26.809926 8257 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:10:26.810060 8257 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:10:26.810571 8257 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:10:26.810616 8257 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:26.810645 8257 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:10:26.810662 8257 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 01:10:26 dist-test-slave-7b22 krb5kdc[7860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385026, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 01:10:26.817083 8257 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45903
I20260926 01:10:26.817152 8387 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45903 every 8 connection(s)
I20260926 01:10:26.817483 8257 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384993574762-392-0/minicluster-data/ts-2/data/info.pb
I20260926 01:10:26.819911 7967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:26.811058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38879 (local address 127.0.98.62:35021)
0926 01:10:26.811124 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:26.811127 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:26.811317 (+ 190us) server_negotiation.cc:407] Connection header received
0926 01:10:26.811845 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:26.811846 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:26.811860 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:26.811933 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:26.812811 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:26.813564 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:26.814767 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:26.814977 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:26.815035 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:26.817238 (+ 2203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:10:26.817239 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:10:26.817240 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:10:26.817249 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:10:26.818710 (+ 1461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:26.819094 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:26.819095 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:26.819095 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:26.819152 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:26.819359 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:26.819359 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:26.819359 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:26.819464 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:10:26.819510 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:26.819774 (+ 264us) server_negotiation.cc:299] Negotiation successful
0926 01:10:26.819848 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 01:10:26.819990 8281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:26.811019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35021 (local address 127.0.98.3:38879)
0926 01:10:26.811179 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:26.811219 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:26.811738 (+ 519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:26.812016 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:26.812033 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:26.812389 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:10:26.812704 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:26.812711 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:26.813659 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:26.813660 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:26.814679 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:26.814682 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:26.815065 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:26.815066 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:26.815207 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:26.815739 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:10:26.815748 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:10:26.817140 (+ 1392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:10:26.818782 (+ 1642us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:10:26.818787 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:10:26.818789 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:10:26.819008 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:10:26.819210 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:10:26.819211 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:10:26.819211 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:10:26.819313 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:10:26.819507 (+ 194us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:10:26.819509 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:10:26.819622 (+ 113us) client_negotiation.cc:769] Sending connection context
0926 01:10:26.819698 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 01:10:26.819833 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":76,"threads_started":1}
I20260926 01:10:26.820492 8388 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35021
I20260926 01:10:26.820572 8388 heartbeater.cc:464] Registering TS with master...
I20260926 01:10:26.820734 8388 heartbeater.cc:511] Master 127.0.98.62:35021 requested a full tablet report, sending...
I20260926 01:10:26.821144 7905 ts_manager.cc:194] Registered new tserver with Master: 39a6c0acb49147dcaa253562329fdde4 (127.0.98.3:45903)
I20260926 01:10:26.821548 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 8257
I20260926 01:10:26.821584 7905 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:38879
I20260926 01:10:26.821658 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384993574762-392-0/minicluster-data/ts-2/wal/instance
I20260926 01:10:26.821913 392 external_mini_cluster.cc:1428] Setting key 89a80f3372ceefc3d7e25cd7b48e0a85
I20260926 01:10:26.822877 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 01:10:26.827565 7967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:26.821495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46925 (local address 127.0.98.62:35021)
0926 01:10:26.821528 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:26.821530 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:26.821986 (+ 456us) server_negotiation.cc:407] Connection header received
0926 01:10:26.822175 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:26.822176 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:26.822187 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:26.822238 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:26.822637 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:26.823245 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:26.824151 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:26.824391 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:26.824460 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:26.825281 (+ 821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:10:26.825282 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:10:26.825283 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:10:26.825290 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:10:26.826578 (+ 1288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:26.826948 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:26.826949 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:26.826949 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:26.826993 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:26.827179 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:26.827179 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:26.827179 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:26.827270 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:10:26.827312 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:26.827453 (+ 141us) server_negotiation.cc:299] Negotiation successful
0926 01:10:26.827505 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260926 01:10:26.827633 8394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:26.821443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35021 (local address 127.0.98.3:46925)
0926 01:10:26.821921 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:26.821932 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:26.822011 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:26.822303 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:26.822304 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:26.822499 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:10:26.822564 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:26.822566 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:26.823345 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:26.823346 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:26.824056 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:26.824058 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:26.824460 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:26.824460 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:26.824520 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:26.824951 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:10:26.824956 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:10:26.825203 (+ 247us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:10:26.826654 (+ 1451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:10:26.826655 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:10:26.826655 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:10:26.826878 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:10:26.827052 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:10:26.827053 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:10:26.827053 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:10:26.827113 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:10:26.827319 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:10:26.827320 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:10:26.827365 (+ 45us) client_negotiation.cc:769] Sending connection context
0926 01:10:26.827438 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 01:10:26.827533 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":59,"threads_started":1}
Sep 26 01:10:26 dist-test-slave-7b22 krb5kdc[7860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385026, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 01:10:26 dist-test-slave-7b22 krb5kdc[7860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:10:26 dist-test-slave-7b22 krb5kdc[7860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260926 01:10:26.843696 7967 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:10:26.838610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47794 (local address 127.0.98.62:35021)
0926 01:10:26.838664 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:26.838667 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:26.838678 (+ 11us) server_negotiation.cc:407] Connection header received
0926 01:10:26.838718 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:26.838719 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:26.838728 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:26.838772 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:26.839313 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:26.839808 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:26.840507 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:26.840685 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:26.840739 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:26.843612 (+ 2873us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47794: BlockingRecv error: recv got EOF from 127.0.0.1:47794 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
Sep 26 01:10:26 dist-test-slave-7b22 krb5kdc[7860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385026, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 01:10:26.850831 7967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:26.844478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47802 (local address 127.0.98.62:35021)
0926 01:10:26.844526 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:26.844528 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:26.844578 (+ 50us) server_negotiation.cc:407] Connection header received
0926 01:10:26.844648 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:26.844649 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:26.844660 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:26.844707 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:26.845228 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:26.845703 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:26.846408 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:26.846612 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:26.846664 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:26.848637 (+ 1973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:10:26.848639 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:10:26.848639 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:10:26.848646 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:10:26.849881 (+ 1235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:26.850159 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:26.850160 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:26.850161 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:26.850209 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:26.850376 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:26.850377 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:26.850378 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:26.850485 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:10:26.850568 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:26.850664 (+ 96us) server_negotiation.cc:299] Negotiation successful
0926 01:10:26.850736 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260926 01:10:26.852487 7905 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47802:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:10:26.854049 7905 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:10:26.860374 8410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:26.856258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45903 (local address 127.0.0.1:43584)
0926 01:10:26.856369 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:26.856400 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:26.856464 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:26.857014 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:26.857017 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:26.857019 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:10:26.857088 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:26.857091 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:26.858111 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:26.858112 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:26.859242 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:26.859245 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:26.860055 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:26.860056 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:26.860155 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:26.860170 (+ 15us) client_negotiation.cc:769] Sending connection context
0926 01:10:26.860233 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 01:10:26.860298 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":53,"threads_started":1}
I20260926 01:10:26.860385 8411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:26.856393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43584 (local address 127.0.98.3:45903)
0926 01:10:26.856759 (+ 366us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:26.856765 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:26.856775 (+ 10us) server_negotiation.cc:407] Connection header received
0926 01:10:26.856814 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:26.856816 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:26.856831 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:26.856927 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:10:26.857152 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:26.858052 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:26.859324 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:26.859986 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:26.860046 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:26.860122 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:26.860254 (+ 132us) server_negotiation.cc:299] Negotiation successful
0926 01:10:26.860294 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":94,"threads_started":1}
I20260926 01:10:26.860775 8414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:26.856595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44745 (local address 127.0.0.1:46048)
0926 01:10:26.857259 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:26.857267 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:26.857311 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:26.857528 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:26.857529 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:26.857531 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:10:26.857565 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:26.857566 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:26.858711 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:26.858712 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:26.859664 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:26.859667 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:26.860568 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:26.860569 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:26.860625 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:26.860628 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 01:10:26.860670 (+ 42us) client_negotiation.cc:240] Negotiation successful
0926 01:10:26.860708 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":649,"thread_start_us":58,"threads_started":1}
I20260926 01:10:26.860850 8415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:26.856716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46048 (local address 127.0.98.2:44745)
0926 01:10:26.856890 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:26.856894 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:26.857360 (+ 466us) server_negotiation.cc:407] Connection header received
0926 01:10:26.857403 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:26.857405 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:26.857419 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:26.857490 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:10:26.857624 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:26.858646 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:26.859752 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:26.860413 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:26.860468 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:26.860537 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:26.860704 (+ 167us) server_negotiation.cc:299] Negotiation successful
0926 01:10:26.860748 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":90,"threads_started":1}
I20260926 01:10:26.861073 8322 tablet_service.cc:1467] Processing CreateTablet for tablet bfdb57d6aaaa40d38e68f641ed9cf0fa (DEFAULT_TABLE table=test-table [id=faded3220ba94e6ca9181f46537a4ce3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:10:26.861191 8412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:26.856444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46585 (local address 127.0.0.1:49500)
0926 01:10:26.856701 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:26.856709 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:26.856868 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:26.857192 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:26.857192 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:26.857195 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:10:26.857244 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:26.857246 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:26.858539 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:26.858540 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:26.859535 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:26.859537 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:26.860749 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:26.860750 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:26.860803 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:26.860806 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 01:10:26.861090 (+ 284us) client_negotiation.cc:240] Negotiation successful
0926 01:10:26.861130 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"mutex_wait_us":31,"thread_start_us":65,"threads_started":1}
I20260926 01:10:26.861253 8413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:26.856570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49500 (local address 127.0.98.1:46585)
0926 01:10:26.857051 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:26.857055 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:26.857064 (+ 9us) server_negotiation.cc:407] Connection header received
0926 01:10:26.857098 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:26.857100 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:26.857114 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:26.857186 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:10:26.857321 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:26.858447 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:26.859600 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:26.860319 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:26.860384 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:26.861033 (+ 649us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:26.861112 (+ 79us) server_negotiation.cc:299] Negotiation successful
0926 01:10:26.861151 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":76,"threads_started":1}
I20260926 01:10:26.861450 8322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfdb57d6aaaa40d38e68f641ed9cf0fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:10:26.862576 8042 tablet_service.cc:1467] Processing CreateTablet for tablet bfdb57d6aaaa40d38e68f641ed9cf0fa (DEFAULT_TABLE table=test-table [id=faded3220ba94e6ca9181f46537a4ce3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:10:26.862577 8182 tablet_service.cc:1467] Processing CreateTablet for tablet bfdb57d6aaaa40d38e68f641ed9cf0fa (DEFAULT_TABLE table=test-table [id=faded3220ba94e6ca9181f46537a4ce3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:10:26.862932 8182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfdb57d6aaaa40d38e68f641ed9cf0fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:10:26.863101 8042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfdb57d6aaaa40d38e68f641ed9cf0fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:10:26.865172 8416 tablet_bootstrap.cc:492] T bfdb57d6aaaa40d38e68f641ed9cf0fa P 39a6c0acb49147dcaa253562329fdde4: Bootstrap starting.
I20260926 01:10:26.865885 8416 tablet_bootstrap.cc:654] T bfdb57d6aaaa40d38e68f641ed9cf0fa P 39a6c0acb49147dcaa253562329fdde4: Neither blocks nor log segments found. Creating new log.
I20260926 01:10:26.866205 8416 log.cc:824] T bfdb57d6aaaa40d38e68f641ed9cf0fa P 39a6c0acb49147dcaa253562329fdde4: Log is configured to *not* fsync() on all Append() calls
I20260926 01:10:26.866242 8417 tablet_bootstrap.cc:492] T bfdb57d6aaaa40d38e68f641ed9cf0fa P feef267fe1a44b048ebeb322ca16140b: Bootstrap starting.
I20260926 01:10:26.866899 8417 tablet_bootstrap.cc:654] T bfdb57d6aaaa40d38e68f641ed9cf0fa P feef267fe1a44b048ebeb322ca16140b: Neither blocks nor log segments found. Creating new log.
I20260926 01:10:26.866883 8419 tablet_bootstrap.cc:492] T bfdb57d6aaaa40d38e68f641ed9cf0fa P e69c8d1efa23415a8c293981f63d9449: Bootstrap starting.
I20260926 01:10:26.866937 8416 tablet_bootstrap.cc:492] T bfdb57d6aaaa40d38e68f641ed9cf0fa P 39a6c0acb49147dcaa253562329fdde4: No bootstrap required, opened a new log
I20260926 01:10:26.866990 8416 ts_tablet_manager.cc:1397] T bfdb57d6aaaa40d38e68f641ed9cf0fa P 39a6c0acb49147dcaa253562329fdde4: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 01:10:26.867182 8417 log.cc:824] T bfdb57d6aaaa40d38e68f641ed9cf0fa P feef267fe1a44b048ebeb322ca16140b: Log is configured to *not* fsync() on all Append() calls
I20260926 01:10:26.867453 8419 tablet_bootstrap.cc:654] T bfdb57d6aaaa40d38e68f641ed9cf0fa P e69c8d1efa23415a8c293981f63d9449: Neither blocks nor log segments found. Creating new log.
I20260926 01:10:26.867740 8419 log.cc:824] T bfdb57d6aaaa40d38e68f641ed9cf0fa P e69c8d1efa23415a8c293981f63d9449: Log is configured to *not* fsync() on all Append() calls
I20260926 01:10:26.867779 8417 tablet_bootstrap.cc:492] T bfdb57d6aaaa40d38e68f641ed9cf0fa P feef267fe1a44b048ebeb322ca16140b: No bootstrap required, opened a new log
I20260926 01:10:26.867839 8417 ts_tablet_manager.cc:1397] T bfdb57d6aaaa40d38e68f641ed9cf0fa P feef267fe1a44b048ebeb322ca16140b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 01:10:26.868727 8419 tablet_bootstrap.cc:492] T bfdb57d6aaaa40d38e68f641ed9cf0fa P e69c8d1efa23415a8c293981f63d9449: No bootstrap required, opened a new log
I20260926 01:10:26.868796 8419 ts_tablet_manager.cc:1397] T bfdb57d6aaaa40d38e68f641ed9cf0fa P e69c8d1efa23415a8c293981f63d9449: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 01:10:26.868907 8416 raft_consensus.cc:348] T bfdb57d6aaaa40d38e68f641ed9cf0fa P 39a6c0acb49147dcaa253562329fdde4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39a6c0acb49147dcaa253562329fdde4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45903 } } peers { permanent_uuid: "e69c8d1efa23415a8c293981f63d9449" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46585 } } peers { permanent_uuid: "feef267fe1a44b048ebeb322ca16140b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44745 } }
I20260926 01:10:26.869066 8416 raft_consensus.cc:374] T bfdb57d6aaaa40d38e68f641ed9cf0fa P 39a6c0acb49147dcaa253562329fdde4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:10:26.869104 8416 raft_consensus.cc:729] T bfdb57d6aaaa40d38e68f641ed9cf0fa P 39a6c0acb49147dcaa253562329fdde4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39a6c0acb49147dcaa253562329fdde4, State: Initialized, Role: FOLLOWER
I20260926 01:10:26.869210 8416 consensus_queue.cc:260] T bfdb57d6aaaa40d38e68f641ed9cf0fa P 39a6c0acb49147dcaa253562329fdde4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39a6c0acb49147dcaa253562329fdde4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45903 } } peers { permanent_uuid: "e69c8d1efa23415a8c293981f63d9449" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46585 } } peers { permanent_uuid: "feef267fe1a44b048ebeb322ca16140b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44745 } }
I20260926 01:10:26.869467 8416 ts_tablet_manager.cc:1428] T bfdb57d6aaaa40d38e68f641ed9cf0fa P 39a6c0acb49147dcaa253562329fdde4: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260926 01:10:26.869506 8388 heartbeater.cc:503] Master 127.0.98.62:35021 was elected leader, sending a full tablet report...
I20260926 01:10:26.869418 8417 raft_consensus.cc:348] T bfdb57d6aaaa40d38e68f641ed9cf0fa P feef267fe1a44b048ebeb322ca16140b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39a6c0acb49147dcaa253562329fdde4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45903 } } peers { permanent_uuid: "e69c8d1efa23415a8c293981f63d9449" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46585 } } peers { permanent_uuid: "feef267fe1a44b048ebeb322ca16140b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44745 } }
I20260926 01:10:26.869557 8417 raft_consensus.cc:374] T bfdb57d6aaaa40d38e68f641ed9cf0fa P feef267fe1a44b048ebeb322ca16140b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:10:26.869586 8417 raft_consensus.cc:729] T bfdb57d6aaaa40d38e68f641ed9cf0fa P feef267fe1a44b048ebeb322ca16140b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feef267fe1a44b048ebeb322ca16140b, State: Initialized, Role: FOLLOWER
I20260926 01:10:26.869690 8417 consensus_queue.cc:260] T bfdb57d6aaaa40d38e68f641ed9cf0fa P feef267fe1a44b048ebeb322ca16140b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39a6c0acb49147dcaa253562329fdde4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45903 } } peers { permanent_uuid: "e69c8d1efa23415a8c293981f63d9449" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46585 } } peers { permanent_uuid: "feef267fe1a44b048ebeb322ca16140b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44745 } }
I20260926 01:10:26.869958 8417 ts_tablet_manager.cc:1428] T bfdb57d6aaaa40d38e68f641ed9cf0fa P feef267fe1a44b048ebeb322ca16140b: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260926 01:10:26.870069 8248 heartbeater.cc:503] Master 127.0.98.62:35021 was elected leader, sending a full tablet report...
I20260926 01:10:26.870947 8419 raft_consensus.cc:348] T bfdb57d6aaaa40d38e68f641ed9cf0fa P e69c8d1efa23415a8c293981f63d9449 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39a6c0acb49147dcaa253562329fdde4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45903 } } peers { permanent_uuid: "e69c8d1efa23415a8c293981f63d9449" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46585 } } peers { permanent_uuid: "feef267fe1a44b048ebeb322ca16140b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44745 } }
I20260926 01:10:26.871060 8419 raft_consensus.cc:374] T bfdb57d6aaaa40d38e68f641ed9cf0fa P e69c8d1efa23415a8c293981f63d9449 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:10:26.871078 8419 raft_consensus.cc:729] T bfdb57d6aaaa40d38e68f641ed9cf0fa P e69c8d1efa23415a8c293981f63d9449 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e69c8d1efa23415a8c293981f63d9449, State: Initialized, Role: FOLLOWER
I20260926 01:10:26.871177 8419 consensus_queue.cc:260] T bfdb57d6aaaa40d38e68f641ed9cf0fa P e69c8d1efa23415a8c293981f63d9449 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39a6c0acb49147dcaa253562329fdde4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45903 } } peers { permanent_uuid: "e69c8d1efa23415a8c293981f63d9449" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46585 } } peers { permanent_uuid: "feef267fe1a44b048ebeb322ca16140b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44745 } }
I20260926 01:10:26.871397 8419 ts_tablet_manager.cc:1428] T bfdb57d6aaaa40d38e68f641ed9cf0fa P e69c8d1efa23415a8c293981f63d9449: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260926 01:10:26.871428 8108 heartbeater.cc:503] Master 127.0.98.62:35021 was elected leader, sending a full tablet report...
W20260926 01:10:26.893535 8249 tablet.cc:2369] T bfdb57d6aaaa40d38e68f641ed9cf0fa P feef267fe1a44b048ebeb322ca16140b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 01:10:26.952564 8424 raft_consensus.cc:482] T bfdb57d6aaaa40d38e68f641ed9cf0fa P e69c8d1efa23415a8c293981f63d9449 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:10:26.952695 8424 raft_consensus.cc:504] T bfdb57d6aaaa40d38e68f641ed9cf0fa P e69c8d1efa23415a8c293981f63d9449 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39a6c0acb49147dcaa253562329fdde4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45903 } } peers { permanent_uuid: "e69c8d1efa23415a8c293981f63d9449" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46585 } } peers { permanent_uuid: "feef267fe1a44b048ebeb322ca16140b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44745 } }
I20260926 01:10:26.953034 8424 leader_election.cc:288] T bfdb57d6aaaa40d38e68f641ed9cf0fa P e69c8d1efa23415a8c293981f63d9449 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39a6c0acb49147dcaa253562329fdde4 (127.0.98.3:45903), feef267fe1a44b048ebeb322ca16140b (127.0.98.2:44745)
I20260926 01:10:26.955921 8415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:26.953430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37511 (local address 127.0.98.2:44745)
0926 01:10:26.953474 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:26.953476 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:26.953551 (+ 75us) server_negotiation.cc:407] Connection header received
0926 01:10:26.953582 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:26.953583 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:26.953594 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:26.953639 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:10:26.953791 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:26.954275 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:26.955228 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:26.955601 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:26.955660 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:26.955671 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:26.955818 (+ 147us) server_negotiation.cc:299] Negotiation successful
0926 01:10:26.955855 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 01:10:26.955929 8411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:26.953328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49851 (local address 127.0.98.3:45903)
0926 01:10:26.953378 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:26.953380 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:26.953391 (+ 11us) server_negotiation.cc:407] Connection header received
0926 01:10:26.953422 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:26.953423 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:26.953435 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:26.953511 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:10:26.953685 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:26.954266 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:26.955242 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:26.955600 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:26.955668 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:26.955679 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:26.955824 (+ 145us) server_negotiation.cc:299] Negotiation successful
0926 01:10:26.955862 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 01:10:26.955945 8425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:26.953332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44745 (local address 127.0.98.1:37511)
0926 01:10:26.953475 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:26.953482 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:26.953529 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:26.953651 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:26.953652 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:26.953653 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:10:26.953728 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:26.953729 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:26.954348 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:26.954350 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:26.955147 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:26.955148 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:26.955673 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:26.955674 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:26.955757 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:26.955761 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 01:10:26.955804 (+ 43us) client_negotiation.cc:240] Negotiation successful
0926 01:10:26.955841 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"mutex_wait_us":1,"thread_start_us":103,"threads_started":1}
I20260926 01:10:26.955956 8001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:26.953224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45903 (local address 127.0.98.1:49851)
0926 01:10:26.953288 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:26.953300 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:26.953345 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:26.953529 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:26.953530 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:26.953533 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:10:26.953610 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:26.953612 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:26.954354 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:26.954355 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:26.955145 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:26.955147 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:26.955675 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:26.955676 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:26.955758 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:26.955761 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 01:10:26.955812 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 01:10:26.955854 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":32}
I20260926 01:10:26.956322 8202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfdb57d6aaaa40d38e68f641ed9cf0fa" candidate_uuid: "e69c8d1efa23415a8c293981f63d9449" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feef267fe1a44b048ebeb322ca16140b" is_pre_election: true
I20260926 01:10:26.956475 8202 raft_consensus.cc:2454] T bfdb57d6aaaa40d38e68f641ed9cf0fa P feef267fe1a44b048ebeb322ca16140b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e69c8d1efa23415a8c293981f63d9449 in term 0.
I20260926 01:10:26.956487 8334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfdb57d6aaaa40d38e68f641ed9cf0fa" candidate_uuid: "e69c8d1efa23415a8c293981f63d9449" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39a6c0acb49147dcaa253562329fdde4" is_pre_election: true
I20260926 01:10:26.956640 8334 raft_consensus.cc:2454] T bfdb57d6aaaa40d38e68f641ed9cf0fa P 39a6c0acb49147dcaa253562329fdde4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e69c8d1efa23415a8c293981f63d9449 in term 0.
I20260926 01:10:26.956854 7994 leader_election.cc:302] T bfdb57d6aaaa40d38e68f641ed9cf0fa P e69c8d1efa23415a8c293981f63d9449 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e69c8d1efa23415a8c293981f63d9449, feef267fe1a44b048ebeb322ca16140b; no voters:
I20260926 01:10:26.956976 8424 raft_consensus.cc:2781] T bfdb57d6aaaa40d38e68f641ed9cf0fa P e69c8d1efa23415a8c293981f63d9449 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:10:26.957042 8424 raft_consensus.cc:482] T bfdb57d6aaaa40d38e68f641ed9cf0fa P e69c8d1efa23415a8c293981f63d9449 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:10:26.957069 8424 raft_consensus.cc:3037] T bfdb57d6aaaa40d38e68f641ed9cf0fa P e69c8d1efa23415a8c293981f63d9449 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:10:26.958130 8424 raft_consensus.cc:504] T bfdb57d6aaaa40d38e68f641ed9cf0fa P e69c8d1efa23415a8c293981f63d9449 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39a6c0acb49147dcaa253562329fdde4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45903 } } peers { permanent_uuid: "e69c8d1efa23415a8c293981f63d9449" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46585 } } peers { permanent_uuid: "feef267fe1a44b048ebeb322ca16140b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44745 } }
I20260926 01:10:26.958266 8424 leader_election.cc:288] T bfdb57d6aaaa40d38e68f641ed9cf0fa P e69c8d1efa23415a8c293981f63d9449 [CANDIDATE]: Term 1 election: Requested vote from peers 39a6c0acb49147dcaa253562329fdde4 (127.0.98.3:45903), feef267fe1a44b048ebeb322ca16140b (127.0.98.2:44745)
I20260926 01:10:26.958425 8334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfdb57d6aaaa40d38e68f641ed9cf0fa" candidate_uuid: "e69c8d1efa23415a8c293981f63d9449" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39a6c0acb49147dcaa253562329fdde4"
I20260926 01:10:26.958485 8202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfdb57d6aaaa40d38e68f641ed9cf0fa" candidate_uuid: "e69c8d1efa23415a8c293981f63d9449" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feef267fe1a44b048ebeb322ca16140b"
I20260926 01:10:26.958506 8334 raft_consensus.cc:3037] T bfdb57d6aaaa40d38e68f641ed9cf0fa P 39a6c0acb49147dcaa253562329fdde4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:10:26.958570 8202 raft_consensus.cc:3037] T bfdb57d6aaaa40d38e68f641ed9cf0fa P feef267fe1a44b048ebeb322ca16140b [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:10:26.959194 8334 raft_consensus.cc:2454] T bfdb57d6aaaa40d38e68f641ed9cf0fa P 39a6c0acb49147dcaa253562329fdde4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e69c8d1efa23415a8c293981f63d9449 in term 1.
I20260926 01:10:26.959301 8202 raft_consensus.cc:2454] T bfdb57d6aaaa40d38e68f641ed9cf0fa P feef267fe1a44b048ebeb322ca16140b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e69c8d1efa23415a8c293981f63d9449 in term 1.
I20260926 01:10:26.959362 7994 leader_election.cc:302] T bfdb57d6aaaa40d38e68f641ed9cf0fa P e69c8d1efa23415a8c293981f63d9449 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39a6c0acb49147dcaa253562329fdde4, e69c8d1efa23415a8c293981f63d9449; no voters:
I20260926 01:10:26.959458 8424 raft_consensus.cc:2781] T bfdb57d6aaaa40d38e68f641ed9cf0fa P e69c8d1efa23415a8c293981f63d9449 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:10:26.959627 8424 raft_consensus.cc:686] T bfdb57d6aaaa40d38e68f641ed9cf0fa P e69c8d1efa23415a8c293981f63d9449 [term 1 LEADER]: Becoming Leader. State: Replica: e69c8d1efa23415a8c293981f63d9449, State: Running, Role: LEADER
I20260926 01:10:26.959728 8424 consensus_queue.cc:237] T bfdb57d6aaaa40d38e68f641ed9cf0fa P e69c8d1efa23415a8c293981f63d9449 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39a6c0acb49147dcaa253562329fdde4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45903 } } peers { permanent_uuid: "e69c8d1efa23415a8c293981f63d9449" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46585 } } peers { permanent_uuid: "feef267fe1a44b048ebeb322ca16140b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44745 } }
I20260926 01:10:26.960645 7905 catalog_manager.cc:5495] T bfdb57d6aaaa40d38e68f641ed9cf0fa P e69c8d1efa23415a8c293981f63d9449 reported cstate change: term changed from 0 to 1, leader changed from <none> to e69c8d1efa23415a8c293981f63d9449 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "e69c8d1efa23415a8c293981f63d9449" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39a6c0acb49147dcaa253562329fdde4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45903 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e69c8d1efa23415a8c293981f63d9449" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46585 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "feef267fe1a44b048ebeb322ca16140b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44745 } health_report { overall_health: UNKNOWN } } }
I20260926 01:10:26.976576 8413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:26.974169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49516 (local address 127.0.98.1:46585)
0926 01:10:26.974248 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:26.974250 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:26.974257 (+ 7us) server_negotiation.cc:407] Connection header received
0926 01:10:26.974294 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:26.974295 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:26.974317 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:26.974377 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:10:26.974487 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:26.975008 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:26.975789 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:26.976043 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:26.976100 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:26.976234 (+ 134us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:10:26.976339 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:10:26.976407 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:26.976458 (+ 51us) server_negotiation.cc:299] Negotiation successful
0926 01:10:26.976482 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 01:10:26.978379 8202 raft_consensus.cc:1261] T bfdb57d6aaaa40d38e68f641ed9cf0fa P feef267fe1a44b048ebeb322ca16140b [term 1 FOLLOWER]: Refusing update from remote peer e69c8d1efa23415a8c293981f63d9449: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:10:26.978698 8334 raft_consensus.cc:1261] T bfdb57d6aaaa40d38e68f641ed9cf0fa P 39a6c0acb49147dcaa253562329fdde4 [term 1 FOLLOWER]: Refusing update from remote peer e69c8d1efa23415a8c293981f63d9449: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:10:26.978698 8424 consensus_queue.cc:1035] T bfdb57d6aaaa40d38e68f641ed9cf0fa P e69c8d1efa23415a8c293981f63d9449 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feef267fe1a44b048ebeb322ca16140b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:10:26.979014 8424 consensus_queue.cc:1035] T bfdb57d6aaaa40d38e68f641ed9cf0fa P e69c8d1efa23415a8c293981f63d9449 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39a6c0acb49147dcaa253562329fdde4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:10:26.980805 8429 mvcc.cc:205] Tried to move back new op lower bound from 7333417070500257792 to 7333417070427648000. Current Snapshot: MvccSnapshot[applied={T|T < 7333417070500257792}]
I20260926 01:10:26.981132 8430 mvcc.cc:205] Tried to move back new op lower bound from 7333417070500257792 to 7333417070427648000. Current Snapshot: MvccSnapshot[applied={T|T < 7333417070500257792}]
I20260926 01:10:26.982999 7904 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47802:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:10:26.983086 7904 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47802:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:10:26.983788 7904 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 142a74e6326f4805a91f83838eeda07e: Sending DeleteTablet for 3 replicas of tablet bfdb57d6aaaa40d38e68f641ed9cf0fa
I20260926 01:10:26.984102 8322 tablet_service.cc:1514] Processing DeleteTablet for tablet bfdb57d6aaaa40d38e68f641ed9cf0fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:10:26 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43584
I20260926 01:10:26.984113 8042 tablet_service.cc:1514] Processing DeleteTablet for tablet bfdb57d6aaaa40d38e68f641ed9cf0fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:10:26 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49500
I20260926 01:10:26.984378 8182 tablet_service.cc:1514] Processing DeleteTablet for tablet bfdb57d6aaaa40d38e68f641ed9cf0fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:10:26 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46048
I20260926 01:10:26.984431 8438 tablet_replica.cc:330] T bfdb57d6aaaa40d38e68f641ed9cf0fa P e69c8d1efa23415a8c293981f63d9449: stopping tablet replica
I20260926 01:10:26.984519 8438 raft_consensus.cc:2229] T bfdb57d6aaaa40d38e68f641ed9cf0fa P e69c8d1efa23415a8c293981f63d9449 [term 1 LEADER]: Raft consensus shutting down.
I20260926 01:10:26.984617 8438 raft_consensus.cc:2258] T bfdb57d6aaaa40d38e68f641ed9cf0fa P e69c8d1efa23415a8c293981f63d9449 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:10:26.984632 7904 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47802:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 01:10:26.984761 8439 tablet_replica.cc:330] T bfdb57d6aaaa40d38e68f641ed9cf0fa P 39a6c0acb49147dcaa253562329fdde4: stopping tablet replica
W20260926 01:10:26.984782 7904 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:10:26.984843 8439 raft_consensus.cc:2229] T bfdb57d6aaaa40d38e68f641ed9cf0fa P 39a6c0acb49147dcaa253562329fdde4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:10:26.984944 8439 raft_consensus.cc:2258] T bfdb57d6aaaa40d38e68f641ed9cf0fa P 39a6c0acb49147dcaa253562329fdde4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:10:26.985389 8440 tablet_replica.cc:330] T bfdb57d6aaaa40d38e68f641ed9cf0fa P feef267fe1a44b048ebeb322ca16140b: stopping tablet replica
I20260926 01:10:26.985592 8438 ts_tablet_manager.cc:1905] T bfdb57d6aaaa40d38e68f641ed9cf0fa P e69c8d1efa23415a8c293981f63d9449: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:10:26.985777 8440 raft_consensus.cc:2229] T bfdb57d6aaaa40d38e68f641ed9cf0fa P feef267fe1a44b048ebeb322ca16140b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:10:26.986060 8042 tablet_service.cc:1467] Processing CreateTablet for tablet fcddaaffb39a4a39a74b592bf127016a (DEFAULT_TABLE table=test-table [id=d1961f2137954d58a75d2876652430bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:10:26.986061 8322 tablet_service.cc:1467] Processing CreateTablet for tablet fcddaaffb39a4a39a74b592bf127016a (DEFAULT_TABLE table=test-table [id=d1961f2137954d58a75d2876652430bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:10:26.986068 8182 tablet_service.cc:1467] Processing CreateTablet for tablet fcddaaffb39a4a39a74b592bf127016a (DEFAULT_TABLE table=test-table [id=d1961f2137954d58a75d2876652430bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:10:26.986161 8322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fcddaaffb39a4a39a74b592bf127016a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:10:26.986181 8042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fcddaaffb39a4a39a74b592bf127016a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:10:26.986187 8182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fcddaaffb39a4a39a74b592bf127016a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:10:26.987457 8419 tablet_bootstrap.cc:492] T fcddaaffb39a4a39a74b592bf127016a P e69c8d1efa23415a8c293981f63d9449: Bootstrap starting.
I20260926 01:10:26.987850 8419 tablet_bootstrap.cc:654] T fcddaaffb39a4a39a74b592bf127016a P e69c8d1efa23415a8c293981f63d9449: Neither blocks nor log segments found. Creating new log.
I20260926 01:10:26.987845 8417 tablet_bootstrap.cc:492] T fcddaaffb39a4a39a74b592bf127016a P feef267fe1a44b048ebeb322ca16140b: Bootstrap starting.
I20260926 01:10:26.988245 8417 tablet_bootstrap.cc:654] T fcddaaffb39a4a39a74b592bf127016a P feef267fe1a44b048ebeb322ca16140b: Neither blocks nor log segments found. Creating new log.
I20260926 01:10:26.988538 8419 tablet_bootstrap.cc:492] T fcddaaffb39a4a39a74b592bf127016a P e69c8d1efa23415a8c293981f63d9449: No bootstrap required, opened a new log
I20260926 01:10:26.988590 8419 ts_tablet_manager.cc:1397] T fcddaaffb39a4a39a74b592bf127016a P e69c8d1efa23415a8c293981f63d9449: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260926 01:10:26.988592 8416 tablet_bootstrap.cc:492] T fcddaaffb39a4a39a74b592bf127016a P 39a6c0acb49147dcaa253562329fdde4: Bootstrap starting.
I20260926 01:10:26.988721 8419 raft_consensus.cc:348] T fcddaaffb39a4a39a74b592bf127016a P e69c8d1efa23415a8c293981f63d9449 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e69c8d1efa23415a8c293981f63d9449" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46585 } } peers { permanent_uuid: "feef267fe1a44b048ebeb322ca16140b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44745 } } peers { permanent_uuid: "39a6c0acb49147dcaa253562329fdde4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45903 } }
I20260926 01:10:26.988804 8419 raft_consensus.cc:374] T fcddaaffb39a4a39a74b592bf127016a P e69c8d1efa23415a8c293981f63d9449 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:10:26.988831 8419 raft_consensus.cc:729] T fcddaaffb39a4a39a74b592bf127016a P e69c8d1efa23415a8c293981f63d9449 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e69c8d1efa23415a8c293981f63d9449, State: Initialized, Role: FOLLOWER
I20260926 01:10:26.988880 8419 consensus_queue.cc:260] T fcddaaffb39a4a39a74b592bf127016a P e69c8d1efa23415a8c293981f63d9449 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e69c8d1efa23415a8c293981f63d9449" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46585 } } peers { permanent_uuid: "feef267fe1a44b048ebeb322ca16140b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44745 } } peers { permanent_uuid: "39a6c0acb49147dcaa253562329fdde4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45903 } }
I20260926 01:10:26.988961 8419 ts_tablet_manager.cc:1428] T fcddaaffb39a4a39a74b592bf127016a P e69c8d1efa23415a8c293981f63d9449: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260926 01:10:26.988973 8416 tablet_bootstrap.cc:654] T fcddaaffb39a4a39a74b592bf127016a P 39a6c0acb49147dcaa253562329fdde4: Neither blocks nor log segments found. Creating new log.
I20260926 01:10:26.989807 8417 tablet_bootstrap.cc:492] T fcddaaffb39a4a39a74b592bf127016a P feef267fe1a44b048ebeb322ca16140b: No bootstrap required, opened a new log
I20260926 01:10:26.989876 8417 ts_tablet_manager.cc:1397] T fcddaaffb39a4a39a74b592bf127016a P feef267fe1a44b048ebeb322ca16140b: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 01:10:26.990018 8417 raft_consensus.cc:348] T fcddaaffb39a4a39a74b592bf127016a P feef267fe1a44b048ebeb322ca16140b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e69c8d1efa23415a8c293981f63d9449" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46585 } } peers { permanent_uuid: "feef267fe1a44b048ebeb322ca16140b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44745 } } peers { permanent_uuid: "39a6c0acb49147dcaa253562329fdde4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45903 } }
I20260926 01:10:26.990093 8417 raft_consensus.cc:374] T fcddaaffb39a4a39a74b592bf127016a P feef267fe1a44b048ebeb322ca16140b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:10:26.990118 8417 raft_consensus.cc:729] T fcddaaffb39a4a39a74b592bf127016a P feef267fe1a44b048ebeb322ca16140b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feef267fe1a44b048ebeb322ca16140b, State: Initialized, Role: FOLLOWER
I20260926 01:10:26.990159 8417 consensus_queue.cc:260] T fcddaaffb39a4a39a74b592bf127016a P feef267fe1a44b048ebeb322ca16140b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e69c8d1efa23415a8c293981f63d9449" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46585 } } peers { permanent_uuid: "feef267fe1a44b048ebeb322ca16140b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44745 } } peers { permanent_uuid: "39a6c0acb49147dcaa253562329fdde4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45903 } }
I20260926 01:10:26.990243 8417 ts_tablet_manager.cc:1428] T fcddaaffb39a4a39a74b592bf127016a P feef267fe1a44b048ebeb322ca16140b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:26.985922 8440 raft_consensus.cc:2258] T bfdb57d6aaaa40d38e68f641ed9cf0fa P feef267fe1a44b048ebeb322ca16140b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:10:26.990532 8439 ts_tablet_manager.cc:1905] T bfdb57d6aaaa40d38e68f641ed9cf0fa P 39a6c0acb49147dcaa253562329fdde4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:10:26.990835 8438 ts_tablet_manager.cc:1918] T bfdb57d6aaaa40d38e68f641ed9cf0fa P e69c8d1efa23415a8c293981f63d9449: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:10:26.990907 8438 log.cc:1196] T bfdb57d6aaaa40d38e68f641ed9cf0fa P e69c8d1efa23415a8c293981f63d9449: Deleting WAL directory at /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384993574762-392-0/minicluster-data/ts-0/wal/wals/bfdb57d6aaaa40d38e68f641ed9cf0fa
I20260926 01:10:26.990945 8440 ts_tablet_manager.cc:1905] T bfdb57d6aaaa40d38e68f641ed9cf0fa P feef267fe1a44b048ebeb322ca16140b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:10:26.991142 8438 ts_tablet_manager.cc:1939] T bfdb57d6aaaa40d38e68f641ed9cf0fa P e69c8d1efa23415a8c293981f63d9449: Deleting consensus metadata
I20260926 01:10:26.991571 7890 catalog_manager.cc:4841] TS e69c8d1efa23415a8c293981f63d9449 (127.0.98.1:46585): tablet bfdb57d6aaaa40d38e68f641ed9cf0fa (table test-table [id=faded3220ba94e6ca9181f46537a4ce3]) successfully deleted
I20260926 01:10:26.992269 8439 ts_tablet_manager.cc:1918] T bfdb57d6aaaa40d38e68f641ed9cf0fa P 39a6c0acb49147dcaa253562329fdde4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:10:26.992388 8440 ts_tablet_manager.cc:1918] T bfdb57d6aaaa40d38e68f641ed9cf0fa P feef267fe1a44b048ebeb322ca16140b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:10:26.992460 8440 log.cc:1196] T bfdb57d6aaaa40d38e68f641ed9cf0fa P feef267fe1a44b048ebeb322ca16140b: Deleting WAL directory at /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384993574762-392-0/minicluster-data/ts-1/wal/wals/bfdb57d6aaaa40d38e68f641ed9cf0fa
I20260926 01:10:26.992414 8439 log.cc:1196] T bfdb57d6aaaa40d38e68f641ed9cf0fa P 39a6c0acb49147dcaa253562329fdde4: Deleting WAL directory at /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384993574762-392-0/minicluster-data/ts-2/wal/wals/bfdb57d6aaaa40d38e68f641ed9cf0fa
I20260926 01:10:26.992488 8416 tablet_bootstrap.cc:492] T fcddaaffb39a4a39a74b592bf127016a P 39a6c0acb49147dcaa253562329fdde4: No bootstrap required, opened a new log
I20260926 01:10:26.992609 8416 ts_tablet_manager.cc:1397] T fcddaaffb39a4a39a74b592bf127016a P 39a6c0acb49147dcaa253562329fdde4: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260926 01:10:26.992698 8440 ts_tablet_manager.cc:1939] T bfdb57d6aaaa40d38e68f641ed9cf0fa P feef267fe1a44b048ebeb322ca16140b: Deleting consensus metadata
I20260926 01:10:26.992763 8416 raft_consensus.cc:348] T fcddaaffb39a4a39a74b592bf127016a P 39a6c0acb49147dcaa253562329fdde4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e69c8d1efa23415a8c293981f63d9449" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46585 } } peers { permanent_uuid: "feef267fe1a44b048ebeb322ca16140b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44745 } } peers { permanent_uuid: "39a6c0acb49147dcaa253562329fdde4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45903 } }
I20260926 01:10:26.992851 8416 raft_consensus.cc:374] T fcddaaffb39a4a39a74b592bf127016a P 39a6c0acb49147dcaa253562329fdde4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:10:26.992875 8416 raft_consensus.cc:729] T fcddaaffb39a4a39a74b592bf127016a P 39a6c0acb49147dcaa253562329fdde4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39a6c0acb49147dcaa253562329fdde4, State: Initialized, Role: FOLLOWER
I20260926 01:10:26.992918 8416 consensus_queue.cc:260] T fcddaaffb39a4a39a74b592bf127016a P 39a6c0acb49147dcaa253562329fdde4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e69c8d1efa23415a8c293981f63d9449" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46585 } } peers { permanent_uuid: "feef267fe1a44b048ebeb322ca16140b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44745 } } peers { permanent_uuid: "39a6c0acb49147dcaa253562329fdde4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45903 } }
I20260926 01:10:26.993000 8416 ts_tablet_manager.cc:1428] T fcddaaffb39a4a39a74b592bf127016a P 39a6c0acb49147dcaa253562329fdde4: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:26.993045 8439 ts_tablet_manager.cc:1939] T bfdb57d6aaaa40d38e68f641ed9cf0fa P 39a6c0acb49147dcaa253562329fdde4: Deleting consensus metadata
I20260926 01:10:26.993163 7890 catalog_manager.cc:4841] TS feef267fe1a44b048ebeb322ca16140b (127.0.98.2:44745): tablet bfdb57d6aaaa40d38e68f641ed9cf0fa (table test-table [id=faded3220ba94e6ca9181f46537a4ce3]) successfully deleted
W20260926 01:10:26.993371 8109 tablet.cc:2369] T fcddaaffb39a4a39a74b592bf127016a P e69c8d1efa23415a8c293981f63d9449: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 01:10:26.993467 7890 catalog_manager.cc:4841] TS 39a6c0acb49147dcaa253562329fdde4 (127.0.98.3:45903): tablet bfdb57d6aaaa40d38e68f641ed9cf0fa (table test-table [id=faded3220ba94e6ca9181f46537a4ce3]) successfully deleted
W20260926 01:10:27.068356 8389 tablet.cc:2369] T fcddaaffb39a4a39a74b592bf127016a P 39a6c0acb49147dcaa253562329fdde4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 01:10:27.069094 8422 raft_consensus.cc:482] T fcddaaffb39a4a39a74b592bf127016a P 39a6c0acb49147dcaa253562329fdde4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:10:27.069165 8422 raft_consensus.cc:504] T fcddaaffb39a4a39a74b592bf127016a P 39a6c0acb49147dcaa253562329fdde4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e69c8d1efa23415a8c293981f63d9449" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46585 } } peers { permanent_uuid: "feef267fe1a44b048ebeb322ca16140b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44745 } } peers { permanent_uuid: "39a6c0acb49147dcaa253562329fdde4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45903 } }
I20260926 01:10:27.069474 8422 leader_election.cc:288] T fcddaaffb39a4a39a74b592bf127016a P 39a6c0acb49147dcaa253562329fdde4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e69c8d1efa23415a8c293981f63d9449 (127.0.98.1:46585), feef267fe1a44b048ebeb322ca16140b (127.0.98.2:44745)
I20260926 01:10:27.073101 8413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:27.069887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56401 (local address 127.0.98.1:46585)
0926 01:10:27.069930 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:27.069932 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:27.069941 (+ 9us) server_negotiation.cc:407] Connection header received
0926 01:10:27.069968 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:27.069969 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:27.069977 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:27.070021 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:10:27.070216 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:27.070935 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:27.072332 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:27.072787 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:27.072844 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:27.072856 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:27.072995 (+ 139us) server_negotiation.cc:299] Negotiation successful
0926 01:10:27.073029 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 01:10:27.073102 8281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:27.069659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46585 (local address 127.0.98.3:56401)
0926 01:10:27.069708 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:27.069720 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:27.069771 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:27.070073 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:27.070074 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:27.070076 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:10:27.070143 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:27.070144 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:27.071021 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:27.071022 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:27.072230 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:27.072233 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:27.072856 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:27.072857 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:27.072932 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:27.072936 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 01:10:27.072981 (+ 45us) client_negotiation.cc:240] Negotiation successful
0926 01:10:27.073029 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260926 01:10:27.073520 8415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:27.069952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38557 (local address 127.0.98.2:44745)
0926 01:10:27.070016 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:27.070018 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:27.070303 (+ 285us) server_negotiation.cc:407] Connection header received
0926 01:10:27.070390 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:27.070391 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:27.070401 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:27.070429 (+ 28us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:10:27.070643 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:27.071301 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:27.072524 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:27.073080 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:27.073172 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:27.073184 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:27.073402 (+ 218us) server_negotiation.cc:299] Negotiation successful
0926 01:10:27.073451 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 01:10:27.073524 8062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcddaaffb39a4a39a74b592bf127016a" candidate_uuid: "39a6c0acb49147dcaa253562329fdde4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e69c8d1efa23415a8c293981f63d9449" is_pre_election: true
I20260926 01:10:27.073652 8062 raft_consensus.cc:2454] T fcddaaffb39a4a39a74b592bf127016a P e69c8d1efa23415a8c293981f63d9449 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39a6c0acb49147dcaa253562329fdde4 in term 0.
I20260926 01:10:27.073807 8444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:27.069997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44745 (local address 127.0.98.3:38557)
0926 01:10:27.070251 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:27.070260 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:27.070325 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:27.070494 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:27.070494 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:27.070496 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:10:27.070578 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:27.070579 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:27.071389 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:27.071390 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:27.072454 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:27.072456 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:27.073220 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:27.073221 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:27.073300 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:27.073305 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 01:10:27.073668 (+ 363us) client_negotiation.cc:240] Negotiation successful
0926 01:10:27.073713 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":65,"threads_started":1}
I20260926 01:10:27.073872 8274 leader_election.cc:302] T fcddaaffb39a4a39a74b592bf127016a P 39a6c0acb49147dcaa253562329fdde4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39a6c0acb49147dcaa253562329fdde4, e69c8d1efa23415a8c293981f63d9449; no voters:
I20260926 01:10:27.074036 8422 raft_consensus.cc:2781] T fcddaaffb39a4a39a74b592bf127016a P 39a6c0acb49147dcaa253562329fdde4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:10:27.074075 8422 raft_consensus.cc:482] T fcddaaffb39a4a39a74b592bf127016a P 39a6c0acb49147dcaa253562329fdde4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:10:27.074095 8422 raft_consensus.cc:3037] T fcddaaffb39a4a39a74b592bf127016a P 39a6c0acb49147dcaa253562329fdde4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:10:27.074143 8202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcddaaffb39a4a39a74b592bf127016a" candidate_uuid: "39a6c0acb49147dcaa253562329fdde4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feef267fe1a44b048ebeb322ca16140b" is_pre_election: true
I20260926 01:10:27.074237 8202 raft_consensus.cc:2454] T fcddaaffb39a4a39a74b592bf127016a P feef267fe1a44b048ebeb322ca16140b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39a6c0acb49147dcaa253562329fdde4 in term 0.
I20260926 01:10:27.074986 8422 raft_consensus.cc:504] T fcddaaffb39a4a39a74b592bf127016a P 39a6c0acb49147dcaa253562329fdde4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e69c8d1efa23415a8c293981f63d9449" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46585 } } peers { permanent_uuid: "feef267fe1a44b048ebeb322ca16140b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44745 } } peers { permanent_uuid: "39a6c0acb49147dcaa253562329fdde4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45903 } }
I20260926 01:10:27.075129 8422 leader_election.cc:288] T fcddaaffb39a4a39a74b592bf127016a P 39a6c0acb49147dcaa253562329fdde4 [CANDIDATE]: Term 1 election: Requested vote from peers e69c8d1efa23415a8c293981f63d9449 (127.0.98.1:46585), feef267fe1a44b048ebeb322ca16140b (127.0.98.2:44745)
I20260926 01:10:27.075362 8202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcddaaffb39a4a39a74b592bf127016a" candidate_uuid: "39a6c0acb49147dcaa253562329fdde4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feef267fe1a44b048ebeb322ca16140b"
I20260926 01:10:27.075380 8062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcddaaffb39a4a39a74b592bf127016a" candidate_uuid: "39a6c0acb49147dcaa253562329fdde4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e69c8d1efa23415a8c293981f63d9449"
I20260926 01:10:27.075444 8202 raft_consensus.cc:3037] T fcddaaffb39a4a39a74b592bf127016a P feef267fe1a44b048ebeb322ca16140b [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:10:27.075455 8062 raft_consensus.cc:3037] T fcddaaffb39a4a39a74b592bf127016a P e69c8d1efa23415a8c293981f63d9449 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:10:27.076064 8202 raft_consensus.cc:2454] T fcddaaffb39a4a39a74b592bf127016a P feef267fe1a44b048ebeb322ca16140b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39a6c0acb49147dcaa253562329fdde4 in term 1.
I20260926 01:10:27.076234 8274 leader_election.cc:302] T fcddaaffb39a4a39a74b592bf127016a P 39a6c0acb49147dcaa253562329fdde4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39a6c0acb49147dcaa253562329fdde4, feef267fe1a44b048ebeb322ca16140b; no voters:
I20260926 01:10:27.076287 8062 raft_consensus.cc:2454] T fcddaaffb39a4a39a74b592bf127016a P e69c8d1efa23415a8c293981f63d9449 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39a6c0acb49147dcaa253562329fdde4 in term 1.
I20260926 01:10:27.076318 8422 raft_consensus.cc:2781] T fcddaaffb39a4a39a74b592bf127016a P 39a6c0acb49147dcaa253562329fdde4 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:10:27.076447 8422 raft_consensus.cc:686] T fcddaaffb39a4a39a74b592bf127016a P 39a6c0acb49147dcaa253562329fdde4 [term 1 LEADER]: Becoming Leader. State: Replica: 39a6c0acb49147dcaa253562329fdde4, State: Running, Role: LEADER
I20260926 01:10:27.076514 8422 consensus_queue.cc:237] T fcddaaffb39a4a39a74b592bf127016a P 39a6c0acb49147dcaa253562329fdde4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e69c8d1efa23415a8c293981f63d9449" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46585 } } peers { permanent_uuid: "feef267fe1a44b048ebeb322ca16140b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44745 } } peers { permanent_uuid: "39a6c0acb49147dcaa253562329fdde4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45903 } }
I20260926 01:10:27.077216 7904 catalog_manager.cc:5495] T fcddaaffb39a4a39a74b592bf127016a P 39a6c0acb49147dcaa253562329fdde4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 39a6c0acb49147dcaa253562329fdde4 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "39a6c0acb49147dcaa253562329fdde4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e69c8d1efa23415a8c293981f63d9449" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "feef267fe1a44b048ebeb322ca16140b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39a6c0acb49147dcaa253562329fdde4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45903 } health_report { overall_health: HEALTHY } } }
Sep 26 01:10:27 dist-test-slave-7b22 krb5kdc[7860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385026, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 01:10:27.105939 7967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:27.098637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47804 (local address 127.0.98.62:35021)
0926 01:10:27.098694 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:27.098696 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:27.098775 (+ 79us) server_negotiation.cc:407] Connection header received
0926 01:10:27.098841 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:27.098842 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:27.098853 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:27.098929 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:27.099347 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:27.100092 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:27.100891 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:27.101156 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:27.101231 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:27.103406 (+ 2175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:10:27.103407 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:10:27.103407 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:10:27.103415 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:10:27.104825 (+ 1410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:27.105153 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:27.105154 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:27.105155 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:27.105201 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:27.105367 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:27.105368 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:27.105368 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:27.105473 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:10:27.105529 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:27.105664 (+ 135us) server_negotiation.cc:299] Negotiation successful
0926 01:10:27.105814 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":74,"server-negotiator.queue_time_us":31}
I20260926 01:10:27.105939 8452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:27.098563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35021 (local address 127.0.0.1:47804)
0926 01:10:27.098701 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:27.098712 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:27.098780 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:27.098934 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:27.098935 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:27.099205 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:10:27.099266 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:27.099268 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:27.100219 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:27.100221 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:27.100774 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:27.100776 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:27.101236 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:27.101237 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:27.101281 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:27.101885 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:10:27.101890 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:10:27.103304 (+ 1414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:10:27.104901 (+ 1597us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:10:27.104909 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:10:27.104911 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:10:27.105093 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:10:27.105265 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:10:27.105265 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:10:27.105266 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:10:27.105320 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:10:27.105525 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:10:27.105533 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:10:27.105579 (+ 46us) client_negotiation.cc:769] Sending connection context
0926 01:10:27.105638 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 01:10:27.105766 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"mutex_wait_us":4,"thread_start_us":69,"threads_started":1}
I20260926 01:10:27.107962 7904 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47804:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 01:10:27.108146 7904 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:10:27.109562 8042 tablet_service.cc:1467] Processing CreateTablet for tablet 396bb7e05136410bbf4cac537c550f8b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e19412b91852439ca02757b45917cdee]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 01:10:27.109627 8322 tablet_service.cc:1467] Processing CreateTablet for tablet 396bb7e05136410bbf4cac537c550f8b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e19412b91852439ca02757b45917cdee]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 01:10:27.109648 8182 tablet_service.cc:1467] Processing CreateTablet for tablet 396bb7e05136410bbf4cac537c550f8b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e19412b91852439ca02757b45917cdee]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 01:10:27.109732 8322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 396bb7e05136410bbf4cac537c550f8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:10:27.109732 8042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 396bb7e05136410bbf4cac537c550f8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:10:27.109792 8182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 396bb7e05136410bbf4cac537c550f8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:10:27.111187 8419 tablet_bootstrap.cc:492] T 396bb7e05136410bbf4cac537c550f8b P e69c8d1efa23415a8c293981f63d9449: Bootstrap starting.
I20260926 01:10:27.111228 8417 tablet_bootstrap.cc:492] T 396bb7e05136410bbf4cac537c550f8b P feef267fe1a44b048ebeb322ca16140b: Bootstrap starting.
I20260926 01:10:27.111479 8419 tablet_bootstrap.cc:654] T 396bb7e05136410bbf4cac537c550f8b P e69c8d1efa23415a8c293981f63d9449: Neither blocks nor log segments found. Creating new log.
I20260926 01:10:27.111596 8417 tablet_bootstrap.cc:654] T 396bb7e05136410bbf4cac537c550f8b P feef267fe1a44b048ebeb322ca16140b: Neither blocks nor log segments found. Creating new log.
I20260926 01:10:27.111694 8416 tablet_bootstrap.cc:492] T 396bb7e05136410bbf4cac537c550f8b P 39a6c0acb49147dcaa253562329fdde4: Bootstrap starting.
I20260926 01:10:27.112071 8416 tablet_bootstrap.cc:654] T 396bb7e05136410bbf4cac537c550f8b P 39a6c0acb49147dcaa253562329fdde4: Neither blocks nor log segments found. Creating new log.
I20260926 01:10:27.112439 8417 tablet_bootstrap.cc:492] T 396bb7e05136410bbf4cac537c550f8b P feef267fe1a44b048ebeb322ca16140b: No bootstrap required, opened a new log
I20260926 01:10:27.112494 8417 ts_tablet_manager.cc:1397] T 396bb7e05136410bbf4cac537c550f8b P feef267fe1a44b048ebeb322ca16140b: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260926 01:10:27.112661 8419 tablet_bootstrap.cc:492] T 396bb7e05136410bbf4cac537c550f8b P e69c8d1efa23415a8c293981f63d9449: No bootstrap required, opened a new log
I20260926 01:10:27.112648 8417 raft_consensus.cc:348] T 396bb7e05136410bbf4cac537c550f8b P feef267fe1a44b048ebeb322ca16140b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e69c8d1efa23415a8c293981f63d9449" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46585 } } peers { permanent_uuid: "feef267fe1a44b048ebeb322ca16140b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44745 } } peers { permanent_uuid: "39a6c0acb49147dcaa253562329fdde4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45903 } }
I20260926 01:10:27.112706 8419 ts_tablet_manager.cc:1397] T 396bb7e05136410bbf4cac537c550f8b P e69c8d1efa23415a8c293981f63d9449: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 01:10:27.112723 8417 raft_consensus.cc:374] T 396bb7e05136410bbf4cac537c550f8b P feef267fe1a44b048ebeb322ca16140b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:10:27.112743 8417 raft_consensus.cc:729] T 396bb7e05136410bbf4cac537c550f8b P feef267fe1a44b048ebeb322ca16140b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feef267fe1a44b048ebeb322ca16140b, State: Initialized, Role: FOLLOWER
I20260926 01:10:27.112793 8417 consensus_queue.cc:260] T 396bb7e05136410bbf4cac537c550f8b P feef267fe1a44b048ebeb322ca16140b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e69c8d1efa23415a8c293981f63d9449" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46585 } } peers { permanent_uuid: "feef267fe1a44b048ebeb322ca16140b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44745 } } peers { permanent_uuid: "39a6c0acb49147dcaa253562329fdde4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45903 } }
I20260926 01:10:27.112821 8419 raft_consensus.cc:348] T 396bb7e05136410bbf4cac537c550f8b P e69c8d1efa23415a8c293981f63d9449 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e69c8d1efa23415a8c293981f63d9449" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46585 } } peers { permanent_uuid: "feef267fe1a44b048ebeb322ca16140b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44745 } } peers { permanent_uuid: "39a6c0acb49147dcaa253562329fdde4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45903 } }
I20260926 01:10:27.112885 8419 raft_consensus.cc:374] T 396bb7e05136410bbf4cac537c550f8b P e69c8d1efa23415a8c293981f63d9449 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:10:27.112882 8417 ts_tablet_manager.cc:1428] T 396bb7e05136410bbf4cac537c550f8b P feef267fe1a44b048ebeb322ca16140b: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260926 01:10:27.112911 8419 raft_consensus.cc:729] T 396bb7e05136410bbf4cac537c550f8b P e69c8d1efa23415a8c293981f63d9449 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e69c8d1efa23415a8c293981f63d9449, State: Initialized, Role: FOLLOWER
I20260926 01:10:27.112951 8419 consensus_queue.cc:260] T 396bb7e05136410bbf4cac537c550f8b P e69c8d1efa23415a8c293981f63d9449 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e69c8d1efa23415a8c293981f63d9449" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46585 } } peers { permanent_uuid: "feef267fe1a44b048ebeb322ca16140b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44745 } } peers { permanent_uuid: "39a6c0acb49147dcaa253562329fdde4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45903 } }
I20260926 01:10:27.113034 8419 ts_tablet_manager.cc:1428] T 396bb7e05136410bbf4cac537c550f8b P e69c8d1efa23415a8c293981f63d9449: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:27.113014 8423 tablet_replica.cc:439] T 396bb7e05136410bbf4cac537c550f8b P feef267fe1a44b048ebeb322ca16140b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e69c8d1efa23415a8c293981f63d9449" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46585 } } peers { permanent_uuid: "feef267fe1a44b048ebeb322ca16140b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44745 } } peers { permanent_uuid: "39a6c0acb49147dcaa253562329fdde4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45903 } } }
I20260926 01:10:27.113104 8423 tablet_replica.cc:442] T 396bb7e05136410bbf4cac537c550f8b P feef267fe1a44b048ebeb322ca16140b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 01:10:27.113198 8424 tablet_replica.cc:439] T 396bb7e05136410bbf4cac537c550f8b P e69c8d1efa23415a8c293981f63d9449: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e69c8d1efa23415a8c293981f63d9449" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46585 } } peers { permanent_uuid: "feef267fe1a44b048ebeb322ca16140b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44745 } } peers { permanent_uuid: "39a6c0acb49147dcaa253562329fdde4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45903 } } }
I20260926 01:10:27.113279 8424 tablet_replica.cc:442] T 396bb7e05136410bbf4cac537c550f8b P e69c8d1efa23415a8c293981f63d9449: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 01:10:27.113925 8416 tablet_bootstrap.cc:492] T 396bb7e05136410bbf4cac537c550f8b P 39a6c0acb49147dcaa253562329fdde4: No bootstrap required, opened a new log
I20260926 01:10:27.113978 8416 ts_tablet_manager.cc:1397] T 396bb7e05136410bbf4cac537c550f8b P 39a6c0acb49147dcaa253562329fdde4: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 01:10:27.114127 8416 raft_consensus.cc:348] T 396bb7e05136410bbf4cac537c550f8b P 39a6c0acb49147dcaa253562329fdde4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e69c8d1efa23415a8c293981f63d9449" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46585 } } peers { permanent_uuid: "feef267fe1a44b048ebeb322ca16140b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44745 } } peers { permanent_uuid: "39a6c0acb49147dcaa253562329fdde4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45903 } }
I20260926 01:10:27.114192 8416 raft_consensus.cc:374] T 396bb7e05136410bbf4cac537c550f8b P 39a6c0acb49147dcaa253562329fdde4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:10:27.114204 8416 raft_consensus.cc:729] T 396bb7e05136410bbf4cac537c550f8b P 39a6c0acb49147dcaa253562329fdde4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39a6c0acb49147dcaa253562329fdde4, State: Initialized, Role: FOLLOWER
I20260926 01:10:27.114238 8416 consensus_queue.cc:260] T 396bb7e05136410bbf4cac537c550f8b P 39a6c0acb49147dcaa253562329fdde4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e69c8d1efa23415a8c293981f63d9449" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46585 } } peers { permanent_uuid: "feef267fe1a44b048ebeb322ca16140b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44745 } } peers { permanent_uuid: "39a6c0acb49147dcaa253562329fdde4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45903 } }
I20260926 01:10:27.114322 8416 ts_tablet_manager.cc:1428] T 396bb7e05136410bbf4cac537c550f8b P 39a6c0acb49147dcaa253562329fdde4: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:27.114431 8422 tablet_replica.cc:439] T 396bb7e05136410bbf4cac537c550f8b P 39a6c0acb49147dcaa253562329fdde4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e69c8d1efa23415a8c293981f63d9449" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46585 } } peers { permanent_uuid: "feef267fe1a44b048ebeb322ca16140b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44745 } } peers { permanent_uuid: "39a6c0acb49147dcaa253562329fdde4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45903 } } }
I20260926 01:10:27.114499 8422 tablet_replica.cc:442] T 396bb7e05136410bbf4cac537c550f8b P 39a6c0acb49147dcaa253562329fdde4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 01:10:27.414798 8423 raft_consensus.cc:482] T 396bb7e05136410bbf4cac537c550f8b P feef267fe1a44b048ebeb322ca16140b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:10:27.414927 8423 raft_consensus.cc:504] T 396bb7e05136410bbf4cac537c550f8b P feef267fe1a44b048ebeb322ca16140b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e69c8d1efa23415a8c293981f63d9449" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46585 } } peers { permanent_uuid: "feef267fe1a44b048ebeb322ca16140b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44745 } } peers { permanent_uuid: "39a6c0acb49147dcaa253562329fdde4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45903 } }
I20260926 01:10:27.415261 8423 leader_election.cc:288] T 396bb7e05136410bbf4cac537c550f8b P feef267fe1a44b048ebeb322ca16140b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e69c8d1efa23415a8c293981f63d9449 (127.0.98.1:46585), 39a6c0acb49147dcaa253562329fdde4 (127.0.98.3:45903)
I20260926 01:10:27.419260 8413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:27.415533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40061 (local address 127.0.98.1:46585)
0926 01:10:27.415588 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:27.415590 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:27.415655 (+ 65us) server_negotiation.cc:407] Connection header received
0926 01:10:27.415720 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:27.415722 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:27.415734 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:27.415814 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:10:27.415976 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:27.416749 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:27.418313 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:27.418878 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:27.418963 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:27.418976 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:27.419140 (+ 164us) server_negotiation.cc:299] Negotiation successful
0926 01:10:27.419178 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 01:10:27.419250 8411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:27.415695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41231 (local address 127.0.98.3:45903)
0926 01:10:27.415742 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:27.415744 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:27.415814 (+ 70us) server_negotiation.cc:407] Connection header received
0926 01:10:27.415924 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:27.415924 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:27.415932 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:27.415978 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:10:27.416090 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:27.416768 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:27.418313 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:27.418938 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:27.419002 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:27.419011 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:27.419135 (+ 124us) server_negotiation.cc:299] Negotiation successful
0926 01:10:27.419175 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 01:10:27.419284 8456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:27.415462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46585 (local address 127.0.98.2:40061)
0926 01:10:27.415582 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:27.415594 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:27.415661 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:27.415827 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:27.415827 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:27.415830 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:10:27.415907 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:27.415910 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:27.416850 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:27.416852 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:27.418197 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:27.418201 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:27.418965 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:27.418967 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:27.419053 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:27.419058 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 01:10:27.419126 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 01:10:27.419177 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"mutex_wait_us":70,"thread_start_us":59,"threads_started":1}
I20260926 01:10:27.419291 8457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:27.415625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45903 (local address 127.0.98.2:41231)
0926 01:10:27.415757 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:27.415765 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:27.415811 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:27.415994 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:27.415995 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:27.415997 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:10:27.416039 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:27.416040 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:27.416864 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:27.416865 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:27.418197 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:27.418201 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:27.419015 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:27.419015 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:27.419074 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:27.419077 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 01:10:27.419126 (+ 49us) client_negotiation.cc:240] Negotiation successful
0926 01:10:27.419177 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"mutex_wait_us":7,"thread_start_us":90,"threads_started":1}
I20260926 01:10:27.419584 8062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "396bb7e05136410bbf4cac537c550f8b" candidate_uuid: "feef267fe1a44b048ebeb322ca16140b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e69c8d1efa23415a8c293981f63d9449" is_pre_election: true
I20260926 01:10:27.419679 8062 raft_consensus.cc:2454] T 396bb7e05136410bbf4cac537c550f8b P e69c8d1efa23415a8c293981f63d9449 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feef267fe1a44b048ebeb322ca16140b in term 0.
I20260926 01:10:27.419826 8334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "396bb7e05136410bbf4cac537c550f8b" candidate_uuid: "feef267fe1a44b048ebeb322ca16140b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39a6c0acb49147dcaa253562329fdde4" is_pre_election: true
I20260926 01:10:27.419839 8134 leader_election.cc:302] T 396bb7e05136410bbf4cac537c550f8b P feef267fe1a44b048ebeb322ca16140b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e69c8d1efa23415a8c293981f63d9449, feef267fe1a44b048ebeb322ca16140b; no voters:
I20260926 01:10:27.419909 8334 raft_consensus.cc:2454] T 396bb7e05136410bbf4cac537c550f8b P 39a6c0acb49147dcaa253562329fdde4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feef267fe1a44b048ebeb322ca16140b in term 0.
I20260926 01:10:27.419962 8423 raft_consensus.cc:2781] T 396bb7e05136410bbf4cac537c550f8b P feef267fe1a44b048ebeb322ca16140b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:10:27.420002 8423 raft_consensus.cc:482] T 396bb7e05136410bbf4cac537c550f8b P feef267fe1a44b048ebeb322ca16140b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:10:27.420022 8423 raft_consensus.cc:3037] T 396bb7e05136410bbf4cac537c550f8b P feef267fe1a44b048ebeb322ca16140b [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:10:27.420758 8423 raft_consensus.cc:504] T 396bb7e05136410bbf4cac537c550f8b P feef267fe1a44b048ebeb322ca16140b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e69c8d1efa23415a8c293981f63d9449" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46585 } } peers { permanent_uuid: "feef267fe1a44b048ebeb322ca16140b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44745 } } peers { permanent_uuid: "39a6c0acb49147dcaa253562329fdde4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45903 } }
I20260926 01:10:27.420878 8423 leader_election.cc:288] T 396bb7e05136410bbf4cac537c550f8b P feef267fe1a44b048ebeb322ca16140b [CANDIDATE]: Term 1 election: Requested vote from peers e69c8d1efa23415a8c293981f63d9449 (127.0.98.1:46585), 39a6c0acb49147dcaa253562329fdde4 (127.0.98.3:45903)
I20260926 01:10:27.421025 8062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "396bb7e05136410bbf4cac537c550f8b" candidate_uuid: "feef267fe1a44b048ebeb322ca16140b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e69c8d1efa23415a8c293981f63d9449"
I20260926 01:10:27.421080 8062 raft_consensus.cc:3037] T 396bb7e05136410bbf4cac537c550f8b P e69c8d1efa23415a8c293981f63d9449 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:10:27.421772 8062 raft_consensus.cc:2454] T 396bb7e05136410bbf4cac537c550f8b P e69c8d1efa23415a8c293981f63d9449 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feef267fe1a44b048ebeb322ca16140b in term 1.
I20260926 01:10:27.421885 8334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "396bb7e05136410bbf4cac537c550f8b" candidate_uuid: "feef267fe1a44b048ebeb322ca16140b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39a6c0acb49147dcaa253562329fdde4"
I20260926 01:10:27.421919 8134 leader_election.cc:302] T 396bb7e05136410bbf4cac537c550f8b P feef267fe1a44b048ebeb322ca16140b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e69c8d1efa23415a8c293981f63d9449, feef267fe1a44b048ebeb322ca16140b; no voters:
I20260926 01:10:27.421957 8334 raft_consensus.cc:3037] T 396bb7e05136410bbf4cac537c550f8b P 39a6c0acb49147dcaa253562329fdde4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:10:27.422001 8423 raft_consensus.cc:2781] T 396bb7e05136410bbf4cac537c550f8b P feef267fe1a44b048ebeb322ca16140b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:10:27.422096 8423 raft_consensus.cc:686] T 396bb7e05136410bbf4cac537c550f8b P feef267fe1a44b048ebeb322ca16140b [term 1 LEADER]: Becoming Leader. State: Replica: feef267fe1a44b048ebeb322ca16140b, State: Running, Role: LEADER
I20260926 01:10:27.422159 8423 consensus_queue.cc:237] T 396bb7e05136410bbf4cac537c550f8b P feef267fe1a44b048ebeb322ca16140b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e69c8d1efa23415a8c293981f63d9449" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46585 } } peers { permanent_uuid: "feef267fe1a44b048ebeb322ca16140b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44745 } } peers { permanent_uuid: "39a6c0acb49147dcaa253562329fdde4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45903 } }
I20260926 01:10:27.422627 8334 raft_consensus.cc:2454] T 396bb7e05136410bbf4cac537c550f8b P 39a6c0acb49147dcaa253562329fdde4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feef267fe1a44b048ebeb322ca16140b in term 1.
I20260926 01:10:27.422622 8458 tablet_replica.cc:439] T 396bb7e05136410bbf4cac537c550f8b P feef267fe1a44b048ebeb322ca16140b: TxnStatusTablet state changed. Reason: New leader feef267fe1a44b048ebeb322ca16140b. Latest consensus state: current_term: 1 leader_uuid: "feef267fe1a44b048ebeb322ca16140b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e69c8d1efa23415a8c293981f63d9449" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46585 } } peers { permanent_uuid: "feef267fe1a44b048ebeb322ca16140b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44745 } } peers { permanent_uuid: "39a6c0acb49147dcaa253562329fdde4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45903 } } }
I20260926 01:10:27.422688 8458 tablet_replica.cc:442] T 396bb7e05136410bbf4cac537c550f8b P feef267fe1a44b048ebeb322ca16140b: This TxnStatusTablet replica's current role is: LEADER
I20260926 01:10:27.422814 7904 catalog_manager.cc:5495] T 396bb7e05136410bbf4cac537c550f8b P feef267fe1a44b048ebeb322ca16140b reported cstate change: term changed from 0 to 1, leader changed from <none> to feef267fe1a44b048ebeb322ca16140b (127.0.98.2). New cstate: current_term: 1 leader_uuid: "feef267fe1a44b048ebeb322ca16140b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e69c8d1efa23415a8c293981f63d9449" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "feef267fe1a44b048ebeb322ca16140b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44745 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39a6c0acb49147dcaa253562329fdde4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45903 } health_report { overall_health: UNKNOWN } } }
I20260926 01:10:27.423221 8460 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 01:10:27.423331 8460 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 01:10:27.468225 8452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:27.465710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44745 (local address 127.0.0.1:46056)
0926 01:10:27.465907 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:27.465920 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:27.465980 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:27.466166 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:27.466167 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:27.466168 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 01:10:27.466220 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:27.466222 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:27.466847 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:27.466848 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:27.467421 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:27.467423 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:27.467736 (+ 313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:27.467736 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:27.467799 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:27.467815 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 01:10:27.468053 (+ 238us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 01:10:27.468060 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 01:10:27.468103 (+ 43us) client_negotiation.cc:240] Negotiation successful
0926 01:10:27.468155 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144}
I20260926 01:10:27.468227 8415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:27.465928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46056 (local address 127.0.98.2:44745)
0926 01:10:27.465974 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:27.465976 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:27.465985 (+ 9us) server_negotiation.cc:407] Connection header received
0926 01:10:27.466061 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:27.466062 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:27.466098 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:27.466152 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:10:27.466305 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:27.466776 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:27.467499 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:27.467678 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:27.467741 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:27.467903 (+ 162us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:10:27.467999 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:10:27.468060 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:27.468107 (+ 47us) server_negotiation.cc:299] Negotiation successful
0926 01:10:27.468128 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 01:10:27.469439 8334 raft_consensus.cc:1261] T 396bb7e05136410bbf4cac537c550f8b P 39a6c0acb49147dcaa253562329fdde4 [term 1 FOLLOWER]: Refusing update from remote peer feef267fe1a44b048ebeb322ca16140b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:10:27.469587 8062 raft_consensus.cc:1261] T 396bb7e05136410bbf4cac537c550f8b P e69c8d1efa23415a8c293981f63d9449 [term 1 FOLLOWER]: Refusing update from remote peer feef267fe1a44b048ebeb322ca16140b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:10:27.469901 8458 consensus_queue.cc:1035] T 396bb7e05136410bbf4cac537c550f8b P feef267fe1a44b048ebeb322ca16140b [LEADER]: Connected to new peer: Peer: permanent_uuid: "39a6c0acb49147dcaa253562329fdde4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:10:27.470027 8423 consensus_queue.cc:1035] T 396bb7e05136410bbf4cac537c550f8b P feef267fe1a44b048ebeb322ca16140b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e69c8d1efa23415a8c293981f63d9449" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:10:27.470671 8424 tablet_replica.cc:439] T 396bb7e05136410bbf4cac537c550f8b P e69c8d1efa23415a8c293981f63d9449: TxnStatusTablet state changed. Reason: New leader feef267fe1a44b048ebeb322ca16140b. Latest consensus state: current_term: 1 leader_uuid: "feef267fe1a44b048ebeb322ca16140b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e69c8d1efa23415a8c293981f63d9449" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46585 } } peers { permanent_uuid: "feef267fe1a44b048ebeb322ca16140b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44745 } } peers { permanent_uuid: "39a6c0acb49147dcaa253562329fdde4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45903 } } }
I20260926 01:10:27.470768 8424 tablet_replica.cc:442] T 396bb7e05136410bbf4cac537c550f8b P e69c8d1efa23415a8c293981f63d9449: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 01:10:27.470829 8422 tablet_replica.cc:439] T 396bb7e05136410bbf4cac537c550f8b P 39a6c0acb49147dcaa253562329fdde4: TxnStatusTablet state changed. Reason: New leader feef267fe1a44b048ebeb322ca16140b. Latest consensus state: current_term: 1 leader_uuid: "feef267fe1a44b048ebeb322ca16140b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e69c8d1efa23415a8c293981f63d9449" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46585 } } peers { permanent_uuid: "feef267fe1a44b048ebeb322ca16140b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44745 } } peers { permanent_uuid: "39a6c0acb49147dcaa253562329fdde4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45903 } } }
I20260926 01:10:27.470903 8422 tablet_replica.cc:442] T 396bb7e05136410bbf4cac537c550f8b P 39a6c0acb49147dcaa253562329fdde4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 01:10:27.471398 8422 tablet_replica.cc:439] T 396bb7e05136410bbf4cac537c550f8b P 39a6c0acb49147dcaa253562329fdde4: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "feef267fe1a44b048ebeb322ca16140b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e69c8d1efa23415a8c293981f63d9449" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46585 } } peers { permanent_uuid: "feef267fe1a44b048ebeb322ca16140b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44745 } } peers { permanent_uuid: "39a6c0acb49147dcaa253562329fdde4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45903 } } }
I20260926 01:10:27.471427 8432 mvcc.cc:205] Tried to move back new op lower bound from 7333417072512942080 to 7333417072321662976. Current Snapshot: MvccSnapshot[applied={T|T < 7333417072512942080}]
I20260926 01:10:27.471475 8422 tablet_replica.cc:442] T 396bb7e05136410bbf4cac537c550f8b P 39a6c0acb49147dcaa253562329fdde4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 01:10:27.471432 8423 tablet_replica.cc:439] T 396bb7e05136410bbf4cac537c550f8b P feef267fe1a44b048ebeb322ca16140b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "feef267fe1a44b048ebeb322ca16140b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e69c8d1efa23415a8c293981f63d9449" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46585 } } peers { permanent_uuid: "feef267fe1a44b048ebeb322ca16140b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44745 } } peers { permanent_uuid: "39a6c0acb49147dcaa253562329fdde4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45903 } } }
I20260926 01:10:27.471504 8423 tablet_replica.cc:442] T 396bb7e05136410bbf4cac537c550f8b P feef267fe1a44b048ebeb322ca16140b: This TxnStatusTablet replica's current role is: LEADER
I20260926 01:10:27.471894 8458 tablet_replica.cc:439] T 396bb7e05136410bbf4cac537c550f8b P feef267fe1a44b048ebeb322ca16140b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "feef267fe1a44b048ebeb322ca16140b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e69c8d1efa23415a8c293981f63d9449" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46585 } } peers { permanent_uuid: "feef267fe1a44b048ebeb322ca16140b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44745 } } peers { permanent_uuid: "39a6c0acb49147dcaa253562329fdde4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45903 } } }
I20260926 01:10:27.472007 8458 tablet_replica.cc:442] T 396bb7e05136410bbf4cac537c550f8b P feef267fe1a44b048ebeb322ca16140b: This TxnStatusTablet replica's current role is: LEADER
I20260926 01:10:27.472360 8424 tablet_replica.cc:439] T 396bb7e05136410bbf4cac537c550f8b P e69c8d1efa23415a8c293981f63d9449: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "feef267fe1a44b048ebeb322ca16140b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e69c8d1efa23415a8c293981f63d9449" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46585 } } peers { permanent_uuid: "feef267fe1a44b048ebeb322ca16140b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44745 } } peers { permanent_uuid: "39a6c0acb49147dcaa253562329fdde4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45903 } } }
I20260926 01:10:27.472450 8424 tablet_replica.cc:442] T 396bb7e05136410bbf4cac537c550f8b P e69c8d1efa23415a8c293981f63d9449: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 01:10:27.476277 8411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:27.473540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43600 (local address 127.0.98.3:45903)
0926 01:10:27.473576 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:27.473578 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:27.473586 (+ 8us) server_negotiation.cc:407] Connection header received
0926 01:10:27.473643 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:27.473644 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:27.473664 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:27.473717 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:10:27.473808 (+ 91us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:27.474516 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:27.475354 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:27.475680 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:27.475747 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:27.475958 (+ 211us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:10:27.476069 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:10:27.476133 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:27.476194 (+ 61us) server_negotiation.cc:299] Negotiation successful
0926 01:10:27.476213 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 01:10:27.481308 8415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:27.478769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51239 (local address 127.0.98.2:44745)
0926 01:10:27.478818 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:27.478820 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:27.478917 (+ 97us) server_negotiation.cc:407] Connection header received
0926 01:10:27.478950 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:27.478950 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:27.478960 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:27.479022 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:10:27.479121 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:27.479802 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:27.480561 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:27.480839 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:27.480913 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:27.481064 (+ 151us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:10:27.481099 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:10:27.481150 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:27.481189 (+ 39us) server_negotiation.cc:299] Negotiation successful
0926 01:10:27.481210 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 01:10:27.481505 8469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:27.478692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44745 (local address 127.0.98.3:51239)
0926 01:10:27.478851 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:27.478862 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:27.478913 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:27.479027 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:27.479028 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:27.479029 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 01:10:27.479074 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:27.479075 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:27.479887 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:27.479888 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:27.480481 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:27.480484 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:27.480931 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:27.480932 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:27.480980 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:27.480993 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 01:10:27.481153 (+ 160us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 01:10:27.481154 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 01:10:27.481183 (+ 29us) client_negotiation.cc:240] Negotiation successful
0926 01:10:27.481214 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":80,"threads_started":1}
I20260926 01:10:27.483886 8062 raft_consensus.cc:1261] T fcddaaffb39a4a39a74b592bf127016a P e69c8d1efa23415a8c293981f63d9449 [term 1 FOLLOWER]: Refusing update from remote peer 39a6c0acb49147dcaa253562329fdde4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:10:27.483901 8202 raft_consensus.cc:1261] T fcddaaffb39a4a39a74b592bf127016a P feef267fe1a44b048ebeb322ca16140b [term 1 FOLLOWER]: Refusing update from remote peer 39a6c0acb49147dcaa253562329fdde4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:10:27.484153 8445 consensus_queue.cc:1035] T fcddaaffb39a4a39a74b592bf127016a P 39a6c0acb49147dcaa253562329fdde4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e69c8d1efa23415a8c293981f63d9449" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:10:27.484298 8422 consensus_queue.cc:1035] T fcddaaffb39a4a39a74b592bf127016a P 39a6c0acb49147dcaa253562329fdde4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feef267fe1a44b048ebeb322ca16140b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:10:27.500579 8411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:27.490158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59907 (local address 127.0.98.3:45903)
0926 01:10:27.490195 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:27.490197 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:27.497998 (+ 7801us) server_negotiation.cc:407] Connection header received
0926 01:10:27.498032 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:27.498033 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:27.498046 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:27.498117 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:10:27.498206 (+ 89us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:27.498964 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:27.499805 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:27.500095 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:27.500155 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:27.500383 (+ 228us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:10:27.500411 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:10:27.500447 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:27.500501 (+ 54us) server_negotiation.cc:299] Negotiation successful
0926 01:10:27.500520 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 01:10:27.500717 8474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:27.489938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45903 (local address 127.0.98.2:59907)
0926 01:10:27.497922 (+ 7984us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:27.497932 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:27.497989 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:27.498117 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:27.498118 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:27.498119 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 01:10:27.498167 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:27.498168 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:27.499040 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:27.499040 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:27.499734 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:27.499738 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:27.500204 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:27.500205 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:27.500271 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:27.500287 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 01:10:27.500451 (+ 164us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 01:10:27.500453 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 01:10:27.500499 (+ 46us) client_negotiation.cc:240] Negotiation successful
0926 01:10:27.500551 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7937,"thread_start_us":84,"threads_started":1}
I20260926 01:10:27.510671 7904 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47802:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:10:27.510792 7904 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47802:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:10:27.511417 7904 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 142a74e6326f4805a91f83838eeda07e: Sending DeleteTablet for 3 replicas of tablet fcddaaffb39a4a39a74b592bf127016a
I20260926 01:10:27.511664 8042 tablet_service.cc:1514] Processing DeleteTablet for tablet fcddaaffb39a4a39a74b592bf127016a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:10:27 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49500
I20260926 01:10:27.511725 8182 tablet_service.cc:1514] Processing DeleteTablet for tablet fcddaaffb39a4a39a74b592bf127016a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:10:27 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46048
I20260926 01:10:27.512027 8478 tablet_replica.cc:330] T fcddaaffb39a4a39a74b592bf127016a P feef267fe1a44b048ebeb322ca16140b: stopping tablet replica
I20260926 01:10:27.512085 8478 raft_consensus.cc:2229] T fcddaaffb39a4a39a74b592bf127016a P feef267fe1a44b048ebeb322ca16140b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:10:27.512120 8478 raft_consensus.cc:2258] T fcddaaffb39a4a39a74b592bf127016a P feef267fe1a44b048ebeb322ca16140b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:10:27.512099 8322 tablet_service.cc:1514] Processing DeleteTablet for tablet fcddaaffb39a4a39a74b592bf127016a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:10:27 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43584
I20260926 01:10:27.512384 8478 ts_tablet_manager.cc:1905] T fcddaaffb39a4a39a74b592bf127016a P feef267fe1a44b048ebeb322ca16140b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:10:27.514086 8478 ts_tablet_manager.cc:1918] T fcddaaffb39a4a39a74b592bf127016a P feef267fe1a44b048ebeb322ca16140b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 01:10:27.514144 8478 log.cc:1196] T fcddaaffb39a4a39a74b592bf127016a P feef267fe1a44b048ebeb322ca16140b: Deleting WAL directory at /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384993574762-392-0/minicluster-data/ts-1/wal/wals/fcddaaffb39a4a39a74b592bf127016a
I20260926 01:10:27.514379 8478 ts_tablet_manager.cc:1939] T fcddaaffb39a4a39a74b592bf127016a P feef267fe1a44b048ebeb322ca16140b: Deleting consensus metadata
I20260926 01:10:27.514531 8477 tablet_replica.cc:330] T fcddaaffb39a4a39a74b592bf127016a P e69c8d1efa23415a8c293981f63d9449: stopping tablet replica
I20260926 01:10:27.514599 8477 raft_consensus.cc:2229] T fcddaaffb39a4a39a74b592bf127016a P e69c8d1efa23415a8c293981f63d9449 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:10:27.514642 8477 raft_consensus.cc:2258] T fcddaaffb39a4a39a74b592bf127016a P e69c8d1efa23415a8c293981f63d9449 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:10:27.514896 8477 ts_tablet_manager.cc:1905] T fcddaaffb39a4a39a74b592bf127016a P e69c8d1efa23415a8c293981f63d9449: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:10:27.515344 8479 tablet_replica.cc:330] T fcddaaffb39a4a39a74b592bf127016a P 39a6c0acb49147dcaa253562329fdde4: stopping tablet replica
I20260926 01:10:27.515452 8479 raft_consensus.cc:2229] T fcddaaffb39a4a39a74b592bf127016a P 39a6c0acb49147dcaa253562329fdde4 [term 1 LEADER]: Raft consensus shutting down.
I20260926 01:10:27.515596 8479 raft_consensus.cc:2258] T fcddaaffb39a4a39a74b592bf127016a P 39a6c0acb49147dcaa253562329fdde4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:10:27.515913 8479 ts_tablet_manager.cc:1905] T fcddaaffb39a4a39a74b592bf127016a P 39a6c0acb49147dcaa253562329fdde4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:10:27.516535 8477 ts_tablet_manager.cc:1918] T fcddaaffb39a4a39a74b592bf127016a P e69c8d1efa23415a8c293981f63d9449: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 01:10:27.516602 8477 log.cc:1196] T fcddaaffb39a4a39a74b592bf127016a P e69c8d1efa23415a8c293981f63d9449: Deleting WAL directory at /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384993574762-392-0/minicluster-data/ts-0/wal/wals/fcddaaffb39a4a39a74b592bf127016a
I20260926 01:10:27.516810 8477 ts_tablet_manager.cc:1939] T fcddaaffb39a4a39a74b592bf127016a P e69c8d1efa23415a8c293981f63d9449: Deleting consensus metadata
I20260926 01:10:27.517798 8479 ts_tablet_manager.cc:1918] T fcddaaffb39a4a39a74b592bf127016a P 39a6c0acb49147dcaa253562329fdde4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 01:10:27.517928 8479 log.cc:1196] T fcddaaffb39a4a39a74b592bf127016a P 39a6c0acb49147dcaa253562329fdde4: Deleting WAL directory at /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384993574762-392-0/minicluster-data/ts-2/wal/wals/fcddaaffb39a4a39a74b592bf127016a
I20260926 01:10:27.518201 8479 ts_tablet_manager.cc:1939] T fcddaaffb39a4a39a74b592bf127016a P 39a6c0acb49147dcaa253562329fdde4: Deleting consensus metadata
I20260926 01:10:27.518635 7890 catalog_manager.cc:4841] TS 39a6c0acb49147dcaa253562329fdde4 (127.0.98.3:45903): tablet fcddaaffb39a4a39a74b592bf127016a (table test-table [id=d1961f2137954d58a75d2876652430bf]) successfully deleted
I20260926 01:10:27.519863 7890 catalog_manager.cc:4841] TS e69c8d1efa23415a8c293981f63d9449 (127.0.98.1:46585): tablet fcddaaffb39a4a39a74b592bf127016a (table test-table [id=d1961f2137954d58a75d2876652430bf]) successfully deleted
I20260926 01:10:27.522096 7890 catalog_manager.cc:4841] TS feef267fe1a44b048ebeb322ca16140b (127.0.98.2:44745): tablet fcddaaffb39a4a39a74b592bf127016a (table test-table [id=d1961f2137954d58a75d2876652430bf]) successfully deleted
Sep 26 01:10:27 dist-test-slave-7b22 krb5kdc[7860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385027, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 01:10:27 dist-test-slave-7b22 krb5kdc[7860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:10:27 dist-test-slave-7b22 krb5kdc[7860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260926 01:10:27.543452 7967 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:10:27.536439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47806 (local address 127.0.98.62:35021)
0926 01:10:27.536483 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:27.536485 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:27.536653 (+ 168us) server_negotiation.cc:407] Connection header received
0926 01:10:27.536714 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:27.536715 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:27.536729 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:27.536790 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:27.537527 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:27.538386 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:27.539517 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:27.539810 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:27.539888 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:27.543384 (+ 3496us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47806: BlockingRecv error: recv got EOF from 127.0.0.1:47806 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Sep 26 01:10:27 dist-test-slave-7b22 krb5kdc[7860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385027, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 01:10:27.552819 7967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:27.544381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47812 (local address 127.0.98.62:35021)
0926 01:10:27.544432 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:27.544434 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:27.544601 (+ 167us) server_negotiation.cc:407] Connection header received
0926 01:10:27.544652 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:27.544653 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:27.544665 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:27.544722 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:27.545279 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:27.546106 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:27.547116 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:27.547427 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:27.547495 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:27.550193 (+ 2698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:10:27.550194 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:10:27.550195 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:10:27.550203 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:10:27.551712 (+ 1509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:27.552083 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:27.552084 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:27.552085 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:27.552143 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:27.552312 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:27.552312 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:27.552312 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:27.552446 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:10:27.552533 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:27.552646 (+ 113us) server_negotiation.cc:299] Negotiation successful
0926 01:10:27.552721 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 01:10:27.556170 7904 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47812:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:10:27.556362 7904 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:10:27.557720 8182 tablet_service.cc:1467] Processing CreateTablet for tablet 92aa1a6ab1ed4055ad226e92ee2a5a1f (DEFAULT_TABLE table=test-table [id=4245d9280c8d4564993d079313722709]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:10:27.557888 8182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92aa1a6ab1ed4055ad226e92ee2a5a1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:10:27.557898 8322 tablet_service.cc:1467] Processing CreateTablet for tablet 92aa1a6ab1ed4055ad226e92ee2a5a1f (DEFAULT_TABLE table=test-table [id=4245d9280c8d4564993d079313722709]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:10:27.558017 8322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92aa1a6ab1ed4055ad226e92ee2a5a1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:10:27.559146 8417 tablet_bootstrap.cc:492] T 92aa1a6ab1ed4055ad226e92ee2a5a1f P feef267fe1a44b048ebeb322ca16140b: Bootstrap starting.
I20260926 01:10:27.559297 8416 tablet_bootstrap.cc:492] T 92aa1a6ab1ed4055ad226e92ee2a5a1f P 39a6c0acb49147dcaa253562329fdde4: Bootstrap starting.
I20260926 01:10:27.559460 8417 tablet_bootstrap.cc:654] T 92aa1a6ab1ed4055ad226e92ee2a5a1f P feef267fe1a44b048ebeb322ca16140b: Neither blocks nor log segments found. Creating new log.
I20260926 01:10:27.559705 8416 tablet_bootstrap.cc:654] T 92aa1a6ab1ed4055ad226e92ee2a5a1f P 39a6c0acb49147dcaa253562329fdde4: Neither blocks nor log segments found. Creating new log.
I20260926 01:10:27.559929 8417 tablet_bootstrap.cc:492] T 92aa1a6ab1ed4055ad226e92ee2a5a1f P feef267fe1a44b048ebeb322ca16140b: No bootstrap required, opened a new log
I20260926 01:10:27.559968 8417 ts_tablet_manager.cc:1397] T 92aa1a6ab1ed4055ad226e92ee2a5a1f P feef267fe1a44b048ebeb322ca16140b: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260926 01:10:27.560063 8417 raft_consensus.cc:348] T 92aa1a6ab1ed4055ad226e92ee2a5a1f P feef267fe1a44b048ebeb322ca16140b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e69c8d1efa23415a8c293981f63d9449" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46585 } } peers { permanent_uuid: "feef267fe1a44b048ebeb322ca16140b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44745 } } peers { permanent_uuid: "39a6c0acb49147dcaa253562329fdde4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45903 } }
I20260926 01:10:27.557789 8042 tablet_service.cc:1467] Processing CreateTablet for tablet 92aa1a6ab1ed4055ad226e92ee2a5a1f (DEFAULT_TABLE table=test-table [id=4245d9280c8d4564993d079313722709]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:10:27.560124 8417 raft_consensus.cc:374] T 92aa1a6ab1ed4055ad226e92ee2a5a1f P feef267fe1a44b048ebeb322ca16140b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:10:27.560155 8417 raft_consensus.cc:729] T 92aa1a6ab1ed4055ad226e92ee2a5a1f P feef267fe1a44b048ebeb322ca16140b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feef267fe1a44b048ebeb322ca16140b, State: Initialized, Role: FOLLOWER
I20260926 01:10:27.560195 8042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92aa1a6ab1ed4055ad226e92ee2a5a1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:10:27.560196 8417 consensus_queue.cc:260] T 92aa1a6ab1ed4055ad226e92ee2a5a1f P feef267fe1a44b048ebeb322ca16140b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e69c8d1efa23415a8c293981f63d9449" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46585 } } peers { permanent_uuid: "feef267fe1a44b048ebeb322ca16140b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44745 } } peers { permanent_uuid: "39a6c0acb49147dcaa253562329fdde4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45903 } }
I20260926 01:10:27.560276 8417 ts_tablet_manager.cc:1428] T 92aa1a6ab1ed4055ad226e92ee2a5a1f P feef267fe1a44b048ebeb322ca16140b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:27.561622 8419 tablet_bootstrap.cc:492] T 92aa1a6ab1ed4055ad226e92ee2a5a1f P e69c8d1efa23415a8c293981f63d9449: Bootstrap starting.
I20260926 01:10:27.561954 8419 tablet_bootstrap.cc:654] T 92aa1a6ab1ed4055ad226e92ee2a5a1f P e69c8d1efa23415a8c293981f63d9449: Neither blocks nor log segments found. Creating new log.
I20260926 01:10:27.562752 8416 tablet_bootstrap.cc:492] T 92aa1a6ab1ed4055ad226e92ee2a5a1f P 39a6c0acb49147dcaa253562329fdde4: No bootstrap required, opened a new log
I20260926 01:10:27.562806 8416 ts_tablet_manager.cc:1397] T 92aa1a6ab1ed4055ad226e92ee2a5a1f P 39a6c0acb49147dcaa253562329fdde4: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260926 01:10:27.562935 8416 raft_consensus.cc:348] T 92aa1a6ab1ed4055ad226e92ee2a5a1f P 39a6c0acb49147dcaa253562329fdde4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e69c8d1efa23415a8c293981f63d9449" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46585 } } peers { permanent_uuid: "feef267fe1a44b048ebeb322ca16140b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44745 } } peers { permanent_uuid: "39a6c0acb49147dcaa253562329fdde4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45903 } }
I20260926 01:10:27.563010 8416 raft_consensus.cc:374] T 92aa1a6ab1ed4055ad226e92ee2a5a1f P 39a6c0acb49147dcaa253562329fdde4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:10:27.563028 8416 raft_consensus.cc:729] T 92aa1a6ab1ed4055ad226e92ee2a5a1f P 39a6c0acb49147dcaa253562329fdde4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39a6c0acb49147dcaa253562329fdde4, State: Initialized, Role: FOLLOWER
I20260926 01:10:27.563066 8416 consensus_queue.cc:260] T 92aa1a6ab1ed4055ad226e92ee2a5a1f P 39a6c0acb49147dcaa253562329fdde4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e69c8d1efa23415a8c293981f63d9449" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46585 } } peers { permanent_uuid: "feef267fe1a44b048ebeb322ca16140b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44745 } } peers { permanent_uuid: "39a6c0acb49147dcaa253562329fdde4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45903 } }
I20260926 01:10:27.563164 8416 ts_tablet_manager.cc:1428] T 92aa1a6ab1ed4055ad226e92ee2a5a1f P 39a6c0acb49147dcaa253562329fdde4: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:27.563421 8419 tablet_bootstrap.cc:492] T 92aa1a6ab1ed4055ad226e92ee2a5a1f P e69c8d1efa23415a8c293981f63d9449: No bootstrap required, opened a new log
I20260926 01:10:27.563464 8419 ts_tablet_manager.cc:1397] T 92aa1a6ab1ed4055ad226e92ee2a5a1f P e69c8d1efa23415a8c293981f63d9449: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 01:10:27.563553 8419 raft_consensus.cc:348] T 92aa1a6ab1ed4055ad226e92ee2a5a1f P e69c8d1efa23415a8c293981f63d9449 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e69c8d1efa23415a8c293981f63d9449" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46585 } } peers { permanent_uuid: "feef267fe1a44b048ebeb322ca16140b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44745 } } peers { permanent_uuid: "39a6c0acb49147dcaa253562329fdde4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45903 } }
I20260926 01:10:27.563612 8419 raft_consensus.cc:374] T 92aa1a6ab1ed4055ad226e92ee2a5a1f P e69c8d1efa23415a8c293981f63d9449 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:10:27.563628 8419 raft_consensus.cc:729] T 92aa1a6ab1ed4055ad226e92ee2a5a1f P e69c8d1efa23415a8c293981f63d9449 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e69c8d1efa23415a8c293981f63d9449, State: Initialized, Role: FOLLOWER
I20260926 01:10:27.563663 8419 consensus_queue.cc:260] T 92aa1a6ab1ed4055ad226e92ee2a5a1f P e69c8d1efa23415a8c293981f63d9449 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e69c8d1efa23415a8c293981f63d9449" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46585 } } peers { permanent_uuid: "feef267fe1a44b048ebeb322ca16140b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44745 } } peers { permanent_uuid: "39a6c0acb49147dcaa253562329fdde4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45903 } }
I20260926 01:10:27.563730 8419 ts_tablet_manager.cc:1428] T 92aa1a6ab1ed4055ad226e92ee2a5a1f P e69c8d1efa23415a8c293981f63d9449: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:27.855682 8424 raft_consensus.cc:482] T 92aa1a6ab1ed4055ad226e92ee2a5a1f P e69c8d1efa23415a8c293981f63d9449 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:10:27.855787 8424 raft_consensus.cc:504] T 92aa1a6ab1ed4055ad226e92ee2a5a1f P e69c8d1efa23415a8c293981f63d9449 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e69c8d1efa23415a8c293981f63d9449" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46585 } } peers { permanent_uuid: "feef267fe1a44b048ebeb322ca16140b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44745 } } peers { permanent_uuid: "39a6c0acb49147dcaa253562329fdde4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45903 } }
I20260926 01:10:27.855943 8424 leader_election.cc:288] T 92aa1a6ab1ed4055ad226e92ee2a5a1f P e69c8d1efa23415a8c293981f63d9449 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers feef267fe1a44b048ebeb322ca16140b (127.0.98.2:44745), 39a6c0acb49147dcaa253562329fdde4 (127.0.98.3:45903)
I20260926 01:10:27.856215 8334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92aa1a6ab1ed4055ad226e92ee2a5a1f" candidate_uuid: "e69c8d1efa23415a8c293981f63d9449" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39a6c0acb49147dcaa253562329fdde4" is_pre_election: true
I20260926 01:10:27.856204 8202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92aa1a6ab1ed4055ad226e92ee2a5a1f" candidate_uuid: "e69c8d1efa23415a8c293981f63d9449" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feef267fe1a44b048ebeb322ca16140b" is_pre_election: true
I20260926 01:10:27.856313 8334 raft_consensus.cc:2454] T 92aa1a6ab1ed4055ad226e92ee2a5a1f P 39a6c0acb49147dcaa253562329fdde4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e69c8d1efa23415a8c293981f63d9449 in term 0.
I20260926 01:10:27.856313 8202 raft_consensus.cc:2454] T 92aa1a6ab1ed4055ad226e92ee2a5a1f P feef267fe1a44b048ebeb322ca16140b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e69c8d1efa23415a8c293981f63d9449 in term 0.
I20260926 01:10:27.856464 7994 leader_election.cc:302] T 92aa1a6ab1ed4055ad226e92ee2a5a1f P e69c8d1efa23415a8c293981f63d9449 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e69c8d1efa23415a8c293981f63d9449, feef267fe1a44b048ebeb322ca16140b; no voters:
I20260926 01:10:27.856585 8424 raft_consensus.cc:2781] T 92aa1a6ab1ed4055ad226e92ee2a5a1f P e69c8d1efa23415a8c293981f63d9449 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:10:27.856633 8424 raft_consensus.cc:482] T 92aa1a6ab1ed4055ad226e92ee2a5a1f P e69c8d1efa23415a8c293981f63d9449 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:10:27.856657 8424 raft_consensus.cc:3037] T 92aa1a6ab1ed4055ad226e92ee2a5a1f P e69c8d1efa23415a8c293981f63d9449 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:10:27.857254 8424 raft_consensus.cc:504] T 92aa1a6ab1ed4055ad226e92ee2a5a1f P e69c8d1efa23415a8c293981f63d9449 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e69c8d1efa23415a8c293981f63d9449" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46585 } } peers { permanent_uuid: "feef267fe1a44b048ebeb322ca16140b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44745 } } peers { permanent_uuid: "39a6c0acb49147dcaa253562329fdde4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45903 } }
I20260926 01:10:27.857349 8424 leader_election.cc:288] T 92aa1a6ab1ed4055ad226e92ee2a5a1f P e69c8d1efa23415a8c293981f63d9449 [CANDIDATE]: Term 1 election: Requested vote from peers feef267fe1a44b048ebeb322ca16140b (127.0.98.2:44745), 39a6c0acb49147dcaa253562329fdde4 (127.0.98.3:45903)
I20260926 01:10:27.857543 8334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92aa1a6ab1ed4055ad226e92ee2a5a1f" candidate_uuid: "e69c8d1efa23415a8c293981f63d9449" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39a6c0acb49147dcaa253562329fdde4"
I20260926 01:10:27.857544 8202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92aa1a6ab1ed4055ad226e92ee2a5a1f" candidate_uuid: "e69c8d1efa23415a8c293981f63d9449" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feef267fe1a44b048ebeb322ca16140b"
I20260926 01:10:27.857616 8334 raft_consensus.cc:3037] T 92aa1a6ab1ed4055ad226e92ee2a5a1f P 39a6c0acb49147dcaa253562329fdde4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:10:27.857616 8202 raft_consensus.cc:3037] T 92aa1a6ab1ed4055ad226e92ee2a5a1f P feef267fe1a44b048ebeb322ca16140b [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:10:27.858389 8334 raft_consensus.cc:2454] T 92aa1a6ab1ed4055ad226e92ee2a5a1f P 39a6c0acb49147dcaa253562329fdde4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e69c8d1efa23415a8c293981f63d9449 in term 1.
I20260926 01:10:27.858395 8202 raft_consensus.cc:2454] T 92aa1a6ab1ed4055ad226e92ee2a5a1f P feef267fe1a44b048ebeb322ca16140b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e69c8d1efa23415a8c293981f63d9449 in term 1.
I20260926 01:10:27.858564 7994 leader_election.cc:302] T 92aa1a6ab1ed4055ad226e92ee2a5a1f P e69c8d1efa23415a8c293981f63d9449 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39a6c0acb49147dcaa253562329fdde4, e69c8d1efa23415a8c293981f63d9449; no voters:
I20260926 01:10:27.858669 8424 raft_consensus.cc:2781] T 92aa1a6ab1ed4055ad226e92ee2a5a1f P e69c8d1efa23415a8c293981f63d9449 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:10:27.858793 8424 raft_consensus.cc:686] T 92aa1a6ab1ed4055ad226e92ee2a5a1f P e69c8d1efa23415a8c293981f63d9449 [term 1 LEADER]: Becoming Leader. State: Replica: e69c8d1efa23415a8c293981f63d9449, State: Running, Role: LEADER
I20260926 01:10:27.858845 8424 consensus_queue.cc:237] T 92aa1a6ab1ed4055ad226e92ee2a5a1f P e69c8d1efa23415a8c293981f63d9449 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e69c8d1efa23415a8c293981f63d9449" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46585 } } peers { permanent_uuid: "feef267fe1a44b048ebeb322ca16140b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44745 } } peers { permanent_uuid: "39a6c0acb49147dcaa253562329fdde4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45903 } }
I20260926 01:10:27.859364 7904 catalog_manager.cc:5495] T 92aa1a6ab1ed4055ad226e92ee2a5a1f P e69c8d1efa23415a8c293981f63d9449 reported cstate change: term changed from 0 to 1, leader changed from <none> to e69c8d1efa23415a8c293981f63d9449 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "e69c8d1efa23415a8c293981f63d9449" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e69c8d1efa23415a8c293981f63d9449" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46585 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "feef267fe1a44b048ebeb322ca16140b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39a6c0acb49147dcaa253562329fdde4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45903 } health_report { overall_health: UNKNOWN } } }
I20260926 01:10:27.919587 8413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:27.917464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49526 (local address 127.0.98.1:46585)
0926 01:10:27.917518 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:27.917520 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:27.917529 (+ 9us) server_negotiation.cc:407] Connection header received
0926 01:10:27.917574 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:27.917575 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:27.917586 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:27.917631 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:10:27.917751 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:27.918252 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:27.919003 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:27.919188 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:27.919241 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:27.919371 (+ 130us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:10:27.919400 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:10:27.919439 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:27.919490 (+ 51us) server_negotiation.cc:299] Negotiation successful
0926 01:10:27.919509 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 01:10:27.920358 8202 raft_consensus.cc:1261] T 92aa1a6ab1ed4055ad226e92ee2a5a1f P feef267fe1a44b048ebeb322ca16140b [term 1 FOLLOWER]: Refusing update from remote peer e69c8d1efa23415a8c293981f63d9449: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:10:27.920364 8334 raft_consensus.cc:1261] T 92aa1a6ab1ed4055ad226e92ee2a5a1f P 39a6c0acb49147dcaa253562329fdde4 [term 1 FOLLOWER]: Refusing update from remote peer e69c8d1efa23415a8c293981f63d9449: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:10:27.920565 8424 consensus_queue.cc:1035] T 92aa1a6ab1ed4055ad226e92ee2a5a1f P e69c8d1efa23415a8c293981f63d9449 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feef267fe1a44b048ebeb322ca16140b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:10:27.920662 8495 consensus_queue.cc:1035] T 92aa1a6ab1ed4055ad226e92ee2a5a1f P e69c8d1efa23415a8c293981f63d9449 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39a6c0acb49147dcaa253562329fdde4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:10:27.923173 7904 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47812:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:10:27.923257 7904 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47812:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:10:27.923775 7904 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 142a74e6326f4805a91f83838eeda07e: Sending DeleteTablet for 3 replicas of tablet 92aa1a6ab1ed4055ad226e92ee2a5a1f
I20260926 01:10:27.923988 8042 tablet_service.cc:1514] Processing DeleteTablet for tablet 92aa1a6ab1ed4055ad226e92ee2a5a1f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:10:27 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49500
I20260926 01:10:27.924002 8182 tablet_service.cc:1514] Processing DeleteTablet for tablet 92aa1a6ab1ed4055ad226e92ee2a5a1f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:10:27 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46048
I20260926 01:10:27.924070 8477 tablet_replica.cc:330] T 92aa1a6ab1ed4055ad226e92ee2a5a1f P e69c8d1efa23415a8c293981f63d9449: stopping tablet replica
I20260926 01:10:27.924090 8478 tablet_replica.cc:330] T 92aa1a6ab1ed4055ad226e92ee2a5a1f P feef267fe1a44b048ebeb322ca16140b: stopping tablet replica
I20260926 01:10:27.924059 8322 tablet_service.cc:1514] Processing DeleteTablet for tablet 92aa1a6ab1ed4055ad226e92ee2a5a1f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:10:27 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43584
I20260926 01:10:27.924117 8477 raft_consensus.cc:2229] T 92aa1a6ab1ed4055ad226e92ee2a5a1f P e69c8d1efa23415a8c293981f63d9449 [term 1 LEADER]: Raft consensus shutting down.
I20260926 01:10:27.924136 8478 raft_consensus.cc:2229] T 92aa1a6ab1ed4055ad226e92ee2a5a1f P feef267fe1a44b048ebeb322ca16140b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:10:27.924162 8479 tablet_replica.cc:330] T 92aa1a6ab1ed4055ad226e92ee2a5a1f P 39a6c0acb49147dcaa253562329fdde4: stopping tablet replica
I20260926 01:10:27.924168 8477 raft_consensus.cc:2258] T 92aa1a6ab1ed4055ad226e92ee2a5a1f P e69c8d1efa23415a8c293981f63d9449 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:10:27.924176 8478 raft_consensus.cc:2258] T 92aa1a6ab1ed4055ad226e92ee2a5a1f P feef267fe1a44b048ebeb322ca16140b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:10:27.924204 8479 raft_consensus.cc:2229] T 92aa1a6ab1ed4055ad226e92ee2a5a1f P 39a6c0acb49147dcaa253562329fdde4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:10:27.924242 8479 raft_consensus.cc:2258] T 92aa1a6ab1ed4055ad226e92ee2a5a1f P 39a6c0acb49147dcaa253562329fdde4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:10:27.924216 7904 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47812:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:10:27.924336 7904 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:10:27.924458 8478 ts_tablet_manager.cc:1905] T 92aa1a6ab1ed4055ad226e92ee2a5a1f P feef267fe1a44b048ebeb322ca16140b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:10:27.924585 8479 ts_tablet_manager.cc:1905] T 92aa1a6ab1ed4055ad226e92ee2a5a1f P 39a6c0acb49147dcaa253562329fdde4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:10:27.924690 8477 ts_tablet_manager.cc:1905] T 92aa1a6ab1ed4055ad226e92ee2a5a1f P e69c8d1efa23415a8c293981f63d9449: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:10:27.925673 8322 tablet_service.cc:1467] Processing CreateTablet for tablet fa6f04b9879a4b01b16c6219aa8c4a0b (DEFAULT_TABLE table=test-table [id=7f0d53b770fe4366ae06a3b3df4c7365]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:10:27.925768 8322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa6f04b9879a4b01b16c6219aa8c4a0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:10:27.926120 8478 ts_tablet_manager.cc:1918] T 92aa1a6ab1ed4055ad226e92ee2a5a1f P feef267fe1a44b048ebeb322ca16140b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:10:27.926169 8478 log.cc:1196] T 92aa1a6ab1ed4055ad226e92ee2a5a1f P feef267fe1a44b048ebeb322ca16140b: Deleting WAL directory at /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384993574762-392-0/minicluster-data/ts-1/wal/wals/92aa1a6ab1ed4055ad226e92ee2a5a1f
I20260926 01:10:27.926199 8477 ts_tablet_manager.cc:1918] T 92aa1a6ab1ed4055ad226e92ee2a5a1f P e69c8d1efa23415a8c293981f63d9449: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:10:27.926208 8479 ts_tablet_manager.cc:1918] T 92aa1a6ab1ed4055ad226e92ee2a5a1f P 39a6c0acb49147dcaa253562329fdde4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:10:27.926240 8477 log.cc:1196] T 92aa1a6ab1ed4055ad226e92ee2a5a1f P e69c8d1efa23415a8c293981f63d9449: Deleting WAL directory at /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384993574762-392-0/minicluster-data/ts-0/wal/wals/92aa1a6ab1ed4055ad226e92ee2a5a1f
I20260926 01:10:27.926254 8479 log.cc:1196] T 92aa1a6ab1ed4055ad226e92ee2a5a1f P 39a6c0acb49147dcaa253562329fdde4: Deleting WAL directory at /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384993574762-392-0/minicluster-data/ts-2/wal/wals/92aa1a6ab1ed4055ad226e92ee2a5a1f
I20260926 01:10:27.926374 8478 ts_tablet_manager.cc:1939] T 92aa1a6ab1ed4055ad226e92ee2a5a1f P feef267fe1a44b048ebeb322ca16140b: Deleting consensus metadata
I20260926 01:10:27.926419 8477 ts_tablet_manager.cc:1939] T 92aa1a6ab1ed4055ad226e92ee2a5a1f P e69c8d1efa23415a8c293981f63d9449: Deleting consensus metadata
I20260926 01:10:27.926466 8479 ts_tablet_manager.cc:1939] T 92aa1a6ab1ed4055ad226e92ee2a5a1f P 39a6c0acb49147dcaa253562329fdde4: Deleting consensus metadata
I20260926 01:10:27.926764 7890 catalog_manager.cc:4841] TS 39a6c0acb49147dcaa253562329fdde4 (127.0.98.3:45903): tablet 92aa1a6ab1ed4055ad226e92ee2a5a1f (table test-table [id=4245d9280c8d4564993d079313722709]) successfully deleted
I20260926 01:10:27.926802 8416 tablet_bootstrap.cc:492] T fa6f04b9879a4b01b16c6219aa8c4a0b P 39a6c0acb49147dcaa253562329fdde4: Bootstrap starting.
I20260926 01:10:27.926872 7890 catalog_manager.cc:4841] TS feef267fe1a44b048ebeb322ca16140b (127.0.98.2:44745): tablet 92aa1a6ab1ed4055ad226e92ee2a5a1f (table test-table [id=4245d9280c8d4564993d079313722709]) successfully deleted
I20260926 01:10:27.926957 8182 tablet_service.cc:1467] Processing CreateTablet for tablet fa6f04b9879a4b01b16c6219aa8c4a0b (DEFAULT_TABLE table=test-table [id=7f0d53b770fe4366ae06a3b3df4c7365]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:10:27.927027 8416 tablet_bootstrap.cc:654] T fa6f04b9879a4b01b16c6219aa8c4a0b P 39a6c0acb49147dcaa253562329fdde4: Neither blocks nor log segments found. Creating new log.
I20260926 01:10:27.927031 8182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa6f04b9879a4b01b16c6219aa8c4a0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:10:27.927423 8416 tablet_bootstrap.cc:492] T fa6f04b9879a4b01b16c6219aa8c4a0b P 39a6c0acb49147dcaa253562329fdde4: No bootstrap required, opened a new log
I20260926 01:10:27.927449 8416 ts_tablet_manager.cc:1397] T fa6f04b9879a4b01b16c6219aa8c4a0b P 39a6c0acb49147dcaa253562329fdde4: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260926 01:10:27.927516 8416 raft_consensus.cc:348] T fa6f04b9879a4b01b16c6219aa8c4a0b P 39a6c0acb49147dcaa253562329fdde4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e69c8d1efa23415a8c293981f63d9449" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46585 } } peers { permanent_uuid: "feef267fe1a44b048ebeb322ca16140b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44745 } } peers { permanent_uuid: "39a6c0acb49147dcaa253562329fdde4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45903 } }
I20260926 01:10:27.927559 8416 raft_consensus.cc:374] T fa6f04b9879a4b01b16c6219aa8c4a0b P 39a6c0acb49147dcaa253562329fdde4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:10:27.927570 8416 raft_consensus.cc:729] T fa6f04b9879a4b01b16c6219aa8c4a0b P 39a6c0acb49147dcaa253562329fdde4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39a6c0acb49147dcaa253562329fdde4, State: Initialized, Role: FOLLOWER
I20260926 01:10:27.927603 8416 consensus_queue.cc:260] T fa6f04b9879a4b01b16c6219aa8c4a0b P 39a6c0acb49147dcaa253562329fdde4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e69c8d1efa23415a8c293981f63d9449" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46585 } } peers { permanent_uuid: "feef267fe1a44b048ebeb322ca16140b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44745 } } peers { permanent_uuid: "39a6c0acb49147dcaa253562329fdde4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45903 } }
I20260926 01:10:27.927666 8416 ts_tablet_manager.cc:1428] T fa6f04b9879a4b01b16c6219aa8c4a0b P 39a6c0acb49147dcaa253562329fdde4: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:27.927987 8417 tablet_bootstrap.cc:492] T fa6f04b9879a4b01b16c6219aa8c4a0b P feef267fe1a44b048ebeb322ca16140b: Bootstrap starting.
I20260926 01:10:27.928238 8417 tablet_bootstrap.cc:654] T fa6f04b9879a4b01b16c6219aa8c4a0b P feef267fe1a44b048ebeb322ca16140b: Neither blocks nor log segments found. Creating new log.
I20260926 01:10:27.928251 7890 catalog_manager.cc:4841] TS e69c8d1efa23415a8c293981f63d9449 (127.0.98.1:46585): tablet 92aa1a6ab1ed4055ad226e92ee2a5a1f (table test-table [id=4245d9280c8d4564993d079313722709]) successfully deleted
I20260926 01:10:27.928315 8042 tablet_service.cc:1467] Processing CreateTablet for tablet fa6f04b9879a4b01b16c6219aa8c4a0b (DEFAULT_TABLE table=test-table [id=7f0d53b770fe4366ae06a3b3df4c7365]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:10:27.928392 8042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa6f04b9879a4b01b16c6219aa8c4a0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:10:27.928653 8417 tablet_bootstrap.cc:492] T fa6f04b9879a4b01b16c6219aa8c4a0b P feef267fe1a44b048ebeb322ca16140b: No bootstrap required, opened a new log
I20260926 01:10:27.928680 8417 ts_tablet_manager.cc:1397] T fa6f04b9879a4b01b16c6219aa8c4a0b P feef267fe1a44b048ebeb322ca16140b: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260926 01:10:27.928747 8417 raft_consensus.cc:348] T fa6f04b9879a4b01b16c6219aa8c4a0b P feef267fe1a44b048ebeb322ca16140b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e69c8d1efa23415a8c293981f63d9449" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46585 } } peers { permanent_uuid: "feef267fe1a44b048ebeb322ca16140b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44745 } } peers { permanent_uuid: "39a6c0acb49147dcaa253562329fdde4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45903 } }
I20260926 01:10:27.928787 8417 raft_consensus.cc:374] T fa6f04b9879a4b01b16c6219aa8c4a0b P feef267fe1a44b048ebeb322ca16140b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:10:27.928798 8417 raft_consensus.cc:729] T fa6f04b9879a4b01b16c6219aa8c4a0b P feef267fe1a44b048ebeb322ca16140b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feef267fe1a44b048ebeb322ca16140b, State: Initialized, Role: FOLLOWER
I20260926 01:10:27.928830 8417 consensus_queue.cc:260] T fa6f04b9879a4b01b16c6219aa8c4a0b P feef267fe1a44b048ebeb322ca16140b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e69c8d1efa23415a8c293981f63d9449" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46585 } } peers { permanent_uuid: "feef267fe1a44b048ebeb322ca16140b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44745 } } peers { permanent_uuid: "39a6c0acb49147dcaa253562329fdde4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45903 } }
I20260926 01:10:27.928891 8417 ts_tablet_manager.cc:1428] T fa6f04b9879a4b01b16c6219aa8c4a0b P feef267fe1a44b048ebeb322ca16140b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:27.929230 8419 tablet_bootstrap.cc:492] T fa6f04b9879a4b01b16c6219aa8c4a0b P e69c8d1efa23415a8c293981f63d9449: Bootstrap starting.
I20260926 01:10:27.929534 8419 tablet_bootstrap.cc:654] T fa6f04b9879a4b01b16c6219aa8c4a0b P e69c8d1efa23415a8c293981f63d9449: Neither blocks nor log segments found. Creating new log.
I20260926 01:10:27.930027 8419 tablet_bootstrap.cc:492] T fa6f04b9879a4b01b16c6219aa8c4a0b P e69c8d1efa23415a8c293981f63d9449: No bootstrap required, opened a new log
I20260926 01:10:27.930075 8419 ts_tablet_manager.cc:1397] T fa6f04b9879a4b01b16c6219aa8c4a0b P e69c8d1efa23415a8c293981f63d9449: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260926 01:10:27.930166 8419 raft_consensus.cc:348] T fa6f04b9879a4b01b16c6219aa8c4a0b P e69c8d1efa23415a8c293981f63d9449 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e69c8d1efa23415a8c293981f63d9449" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46585 } } peers { permanent_uuid: "feef267fe1a44b048ebeb322ca16140b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44745 } } peers { permanent_uuid: "39a6c0acb49147dcaa253562329fdde4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45903 } }
I20260926 01:10:27.930223 8419 raft_consensus.cc:374] T fa6f04b9879a4b01b16c6219aa8c4a0b P e69c8d1efa23415a8c293981f63d9449 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:10:27.930243 8419 raft_consensus.cc:729] T fa6f04b9879a4b01b16c6219aa8c4a0b P e69c8d1efa23415a8c293981f63d9449 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e69c8d1efa23415a8c293981f63d9449, State: Initialized, Role: FOLLOWER
I20260926 01:10:27.930285 8419 consensus_queue.cc:260] T fa6f04b9879a4b01b16c6219aa8c4a0b P e69c8d1efa23415a8c293981f63d9449 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e69c8d1efa23415a8c293981f63d9449" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46585 } } peers { permanent_uuid: "feef267fe1a44b048ebeb322ca16140b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44745 } } peers { permanent_uuid: "39a6c0acb49147dcaa253562329fdde4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45903 } }
I20260926 01:10:27.930372 8419 ts_tablet_manager.cc:1428] T fa6f04b9879a4b01b16c6219aa8c4a0b P e69c8d1efa23415a8c293981f63d9449: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:28.143960 8424 raft_consensus.cc:482] T fa6f04b9879a4b01b16c6219aa8c4a0b P e69c8d1efa23415a8c293981f63d9449 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:10:28.144068 8424 raft_consensus.cc:504] T fa6f04b9879a4b01b16c6219aa8c4a0b P e69c8d1efa23415a8c293981f63d9449 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e69c8d1efa23415a8c293981f63d9449" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46585 } } peers { permanent_uuid: "feef267fe1a44b048ebeb322ca16140b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44745 } } peers { permanent_uuid: "39a6c0acb49147dcaa253562329fdde4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45903 } }
I20260926 01:10:28.144227 8424 leader_election.cc:288] T fa6f04b9879a4b01b16c6219aa8c4a0b P e69c8d1efa23415a8c293981f63d9449 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers feef267fe1a44b048ebeb322ca16140b (127.0.98.2:44745), 39a6c0acb49147dcaa253562329fdde4 (127.0.98.3:45903)
I20260926 01:10:28.144495 8202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa6f04b9879a4b01b16c6219aa8c4a0b" candidate_uuid: "e69c8d1efa23415a8c293981f63d9449" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feef267fe1a44b048ebeb322ca16140b" is_pre_election: true
I20260926 01:10:28.144506 8334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa6f04b9879a4b01b16c6219aa8c4a0b" candidate_uuid: "e69c8d1efa23415a8c293981f63d9449" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39a6c0acb49147dcaa253562329fdde4" is_pre_election: true
I20260926 01:10:28.144590 8202 raft_consensus.cc:2454] T fa6f04b9879a4b01b16c6219aa8c4a0b P feef267fe1a44b048ebeb322ca16140b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e69c8d1efa23415a8c293981f63d9449 in term 0.
I20260926 01:10:28.144608 8334 raft_consensus.cc:2454] T fa6f04b9879a4b01b16c6219aa8c4a0b P 39a6c0acb49147dcaa253562329fdde4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e69c8d1efa23415a8c293981f63d9449 in term 0.
I20260926 01:10:28.144762 7994 leader_election.cc:302] T fa6f04b9879a4b01b16c6219aa8c4a0b P e69c8d1efa23415a8c293981f63d9449 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39a6c0acb49147dcaa253562329fdde4, e69c8d1efa23415a8c293981f63d9449; no voters:
I20260926 01:10:28.144899 8424 raft_consensus.cc:2781] T fa6f04b9879a4b01b16c6219aa8c4a0b P e69c8d1efa23415a8c293981f63d9449 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:10:28.144943 8424 raft_consensus.cc:482] T fa6f04b9879a4b01b16c6219aa8c4a0b P e69c8d1efa23415a8c293981f63d9449 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:10:28.144963 8424 raft_consensus.cc:3037] T fa6f04b9879a4b01b16c6219aa8c4a0b P e69c8d1efa23415a8c293981f63d9449 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:10:28.145593 8424 raft_consensus.cc:504] T fa6f04b9879a4b01b16c6219aa8c4a0b P e69c8d1efa23415a8c293981f63d9449 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e69c8d1efa23415a8c293981f63d9449" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46585 } } peers { permanent_uuid: "feef267fe1a44b048ebeb322ca16140b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44745 } } peers { permanent_uuid: "39a6c0acb49147dcaa253562329fdde4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45903 } }
I20260926 01:10:28.145699 8424 leader_election.cc:288] T fa6f04b9879a4b01b16c6219aa8c4a0b P e69c8d1efa23415a8c293981f63d9449 [CANDIDATE]: Term 1 election: Requested vote from peers feef267fe1a44b048ebeb322ca16140b (127.0.98.2:44745), 39a6c0acb49147dcaa253562329fdde4 (127.0.98.3:45903)
I20260926 01:10:28.145934 8334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa6f04b9879a4b01b16c6219aa8c4a0b" candidate_uuid: "e69c8d1efa23415a8c293981f63d9449" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39a6c0acb49147dcaa253562329fdde4"
I20260926 01:10:28.145908 8202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa6f04b9879a4b01b16c6219aa8c4a0b" candidate_uuid: "e69c8d1efa23415a8c293981f63d9449" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feef267fe1a44b048ebeb322ca16140b"
I20260926 01:10:28.146006 8334 raft_consensus.cc:3037] T fa6f04b9879a4b01b16c6219aa8c4a0b P 39a6c0acb49147dcaa253562329fdde4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:10:28.146011 8202 raft_consensus.cc:3037] T fa6f04b9879a4b01b16c6219aa8c4a0b P feef267fe1a44b048ebeb322ca16140b [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:10:28.146591 8334 raft_consensus.cc:2454] T fa6f04b9879a4b01b16c6219aa8c4a0b P 39a6c0acb49147dcaa253562329fdde4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e69c8d1efa23415a8c293981f63d9449 in term 1.
I20260926 01:10:28.146703 8202 raft_consensus.cc:2454] T fa6f04b9879a4b01b16c6219aa8c4a0b P feef267fe1a44b048ebeb322ca16140b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e69c8d1efa23415a8c293981f63d9449 in term 1.
I20260926 01:10:28.146795 7994 leader_election.cc:302] T fa6f04b9879a4b01b16c6219aa8c4a0b P e69c8d1efa23415a8c293981f63d9449 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39a6c0acb49147dcaa253562329fdde4, e69c8d1efa23415a8c293981f63d9449; no voters:
I20260926 01:10:28.146890 8424 raft_consensus.cc:2781] T fa6f04b9879a4b01b16c6219aa8c4a0b P e69c8d1efa23415a8c293981f63d9449 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:10:28.146945 8424 raft_consensus.cc:686] T fa6f04b9879a4b01b16c6219aa8c4a0b P e69c8d1efa23415a8c293981f63d9449 [term 1 LEADER]: Becoming Leader. State: Replica: e69c8d1efa23415a8c293981f63d9449, State: Running, Role: LEADER
I20260926 01:10:28.146998 8424 consensus_queue.cc:237] T fa6f04b9879a4b01b16c6219aa8c4a0b P e69c8d1efa23415a8c293981f63d9449 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e69c8d1efa23415a8c293981f63d9449" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46585 } } peers { permanent_uuid: "feef267fe1a44b048ebeb322ca16140b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44745 } } peers { permanent_uuid: "39a6c0acb49147dcaa253562329fdde4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45903 } }
I20260926 01:10:28.147554 7904 catalog_manager.cc:5495] T fa6f04b9879a4b01b16c6219aa8c4a0b P e69c8d1efa23415a8c293981f63d9449 reported cstate change: term changed from 0 to 1, leader changed from <none> to e69c8d1efa23415a8c293981f63d9449 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "e69c8d1efa23415a8c293981f63d9449" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e69c8d1efa23415a8c293981f63d9449" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46585 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "feef267fe1a44b048ebeb322ca16140b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39a6c0acb49147dcaa253562329fdde4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45903 } health_report { overall_health: UNKNOWN } } }
I20260926 01:10:28.163268 8508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:28.160746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44745 (local address 127.0.98.1:56337)
0926 01:10:28.160873 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:28.160905 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:28.160956 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:28.161072 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:28.161072 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:28.161073 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 01:10:28.161116 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:28.161117 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:28.161719 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:28.161720 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:28.162414 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:28.162416 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:28.162772 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:28.162772 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:28.162813 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:28.162826 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 01:10:28.163071 (+ 245us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 01:10:28.163073 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 01:10:28.163131 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 01:10:28.163183 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":69,"threads_started":1}
I20260926 01:10:28.163251 8509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:28.160829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56337 (local address 127.0.98.2:44745)
0926 01:10:28.160913 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:28.160915 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:28.160945 (+ 30us) server_negotiation.cc:407] Connection header received
0926 01:10:28.161003 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:28.161004 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:28.161013 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:28.161068 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:10:28.161157 (+ 89us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:28.161636 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:28.162482 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:28.162687 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:28.162762 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:28.162953 (+ 191us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:10:28.162992 (+ 39us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:10:28.163067 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:28.163136 (+ 69us) server_negotiation.cc:299] Negotiation successful
0926 01:10:28.163160 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65,"thread_start_us":45,"threads_started":1}
I20260926 01:10:28.165189 8334 raft_consensus.cc:1261] T fa6f04b9879a4b01b16c6219aa8c4a0b P 39a6c0acb49147dcaa253562329fdde4 [term 1 FOLLOWER]: Refusing update from remote peer e69c8d1efa23415a8c293981f63d9449: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:10:28.165211 8202 raft_consensus.cc:1261] T fa6f04b9879a4b01b16c6219aa8c4a0b P feef267fe1a44b048ebeb322ca16140b [term 1 FOLLOWER]: Refusing update from remote peer e69c8d1efa23415a8c293981f63d9449: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:10:28.165395 8424 consensus_queue.cc:1035] T fa6f04b9879a4b01b16c6219aa8c4a0b P e69c8d1efa23415a8c293981f63d9449 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39a6c0acb49147dcaa253562329fdde4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:10:28.165525 8495 consensus_queue.cc:1035] T fa6f04b9879a4b01b16c6219aa8c4a0b P e69c8d1efa23415a8c293981f63d9449 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feef267fe1a44b048ebeb322ca16140b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:10:28.171844 8413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:28.169476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39467 (local address 127.0.98.1:46585)
0926 01:10:28.169532 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:28.169533 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:28.169600 (+ 67us) server_negotiation.cc:407] Connection header received
0926 01:10:28.169672 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:28.169673 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:28.169684 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:28.169742 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:10:28.169917 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:28.170404 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:28.171143 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:28.171380 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:28.171451 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:28.171580 (+ 129us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:10:28.171615 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:10:28.171661 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:28.171749 (+ 88us) server_negotiation.cc:299] Negotiation successful
0926 01:10:28.171769 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 01:10:28.171859 8514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:28.169417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46585 (local address 127.0.98.2:39467)
0926 01:10:28.169547 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:28.169558 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:28.169615 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:28.169788 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:28.169789 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:28.169791 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 01:10:28.169861 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:28.169863 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:28.170478 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:28.170479 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:28.171061 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:28.171063 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:28.171452 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:28.171452 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:28.171510 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:28.171515 (+ 5us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 01:10:28.171687 (+ 172us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 01:10:28.171688 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 01:10:28.171737 (+ 49us) client_negotiation.cc:240] Negotiation successful
0926 01:10:28.171784 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":79,"threads_started":1}
I20260926 01:10:28.177317 7905 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47812:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:10:28.177415 7905 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47812:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:10:28.178001 7905 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 142a74e6326f4805a91f83838eeda07e: Sending DeleteTablet for 3 replicas of tablet fa6f04b9879a4b01b16c6219aa8c4a0b
I20260926 01:10:28.178289 8042 tablet_service.cc:1514] Processing DeleteTablet for tablet fa6f04b9879a4b01b16c6219aa8c4a0b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:10:28 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49500
I20260926 01:10:28.178300 8182 tablet_service.cc:1514] Processing DeleteTablet for tablet fa6f04b9879a4b01b16c6219aa8c4a0b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:10:28 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46048
I20260926 01:10:28.178330 8322 tablet_service.cc:1514] Processing DeleteTablet for tablet fa6f04b9879a4b01b16c6219aa8c4a0b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:10:28 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43584
I20260926 01:10:28.178390 8477 tablet_replica.cc:330] T fa6f04b9879a4b01b16c6219aa8c4a0b P e69c8d1efa23415a8c293981f63d9449: stopping tablet replica
I20260926 01:10:28.178392 8478 tablet_replica.cc:330] T fa6f04b9879a4b01b16c6219aa8c4a0b P feef267fe1a44b048ebeb322ca16140b: stopping tablet replica
I20260926 01:10:28.178412 8479 tablet_replica.cc:330] T fa6f04b9879a4b01b16c6219aa8c4a0b P 39a6c0acb49147dcaa253562329fdde4: stopping tablet replica
I20260926 01:10:28.178442 8478 raft_consensus.cc:2229] T fa6f04b9879a4b01b16c6219aa8c4a0b P feef267fe1a44b048ebeb322ca16140b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:10:28.178444 8477 raft_consensus.cc:2229] T fa6f04b9879a4b01b16c6219aa8c4a0b P e69c8d1efa23415a8c293981f63d9449 [term 1 LEADER]: Raft consensus shutting down.
I20260926 01:10:28.178458 8479 raft_consensus.cc:2229] T fa6f04b9879a4b01b16c6219aa8c4a0b P 39a6c0acb49147dcaa253562329fdde4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:10:28.178470 8478 raft_consensus.cc:2258] T fa6f04b9879a4b01b16c6219aa8c4a0b P feef267fe1a44b048ebeb322ca16140b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:10:28.178490 8479 raft_consensus.cc:2258] T fa6f04b9879a4b01b16c6219aa8c4a0b P 39a6c0acb49147dcaa253562329fdde4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:10:28.178498 8477 raft_consensus.cc:2258] T fa6f04b9879a4b01b16c6219aa8c4a0b P e69c8d1efa23415a8c293981f63d9449 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:10:28.178848 8478 ts_tablet_manager.cc:1905] T fa6f04b9879a4b01b16c6219aa8c4a0b P feef267fe1a44b048ebeb322ca16140b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:10:28.178894 8479 ts_tablet_manager.cc:1905] T fa6f04b9879a4b01b16c6219aa8c4a0b P 39a6c0acb49147dcaa253562329fdde4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:10:28.178975 8477 ts_tablet_manager.cc:1905] T fa6f04b9879a4b01b16c6219aa8c4a0b P e69c8d1efa23415a8c293981f63d9449: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:10:28.180600 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 7977
I20260926 01:10:28.181159 8478 ts_tablet_manager.cc:1918] T fa6f04b9879a4b01b16c6219aa8c4a0b P feef267fe1a44b048ebeb322ca16140b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 01:10:28.181209 8478 log.cc:1196] T fa6f04b9879a4b01b16c6219aa8c4a0b P feef267fe1a44b048ebeb322ca16140b: Deleting WAL directory at /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384993574762-392-0/minicluster-data/ts-1/wal/wals/fa6f04b9879a4b01b16c6219aa8c4a0b
I20260926 01:10:28.181233 8479 ts_tablet_manager.cc:1918] T fa6f04b9879a4b01b16c6219aa8c4a0b P 39a6c0acb49147dcaa253562329fdde4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 01:10:28.181294 8479 log.cc:1196] T fa6f04b9879a4b01b16c6219aa8c4a0b P 39a6c0acb49147dcaa253562329fdde4: Deleting WAL directory at /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384993574762-392-0/minicluster-data/ts-2/wal/wals/fa6f04b9879a4b01b16c6219aa8c4a0b
I20260926 01:10:28.181420 8478 ts_tablet_manager.cc:1939] T fa6f04b9879a4b01b16c6219aa8c4a0b P feef267fe1a44b048ebeb322ca16140b: Deleting consensus metadata
I20260926 01:10:28.181511 8479 ts_tablet_manager.cc:1939] T fa6f04b9879a4b01b16c6219aa8c4a0b P 39a6c0acb49147dcaa253562329fdde4: Deleting consensus metadata
I20260926 01:10:28.181720 7890 catalog_manager.cc:4841] TS feef267fe1a44b048ebeb322ca16140b (127.0.98.2:44745): tablet fa6f04b9879a4b01b16c6219aa8c4a0b (table test-table [id=7f0d53b770fe4366ae06a3b3df4c7365]) successfully deleted
I20260926 01:10:28.181802 7890 catalog_manager.cc:4841] TS 39a6c0acb49147dcaa253562329fdde4 (127.0.98.3:45903): tablet fa6f04b9879a4b01b16c6219aa8c4a0b (table test-table [id=7f0d53b770fe4366ae06a3b3df4c7365]) successfully deleted
W20260926 01:10:28.187080 7890 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46585 (error 108)
I20260926 01:10:28.187259 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 8117
I20260926 01:10:28.188374 8517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:28.187844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46585 (local address 127.0.0.1:49532)
0926 01:10:28.188304 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:28.188339 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":36,"threads_started":1}
W20260926 01:10:28.188484 7890 catalog_manager.cc:4584] TS e69c8d1efa23415a8c293981f63d9449 (127.0.98.1:46585): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fa6f04b9879a4b01b16c6219aa8c4a0b: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46585: connect: Connection refused (error 111)
I20260926 01:10:28.192317 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 8257
I20260926 01:10:28.196848 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 7873
2026-09-26T01:10:28Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3072 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790384993574762-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 01:10:28 dist-test-slave-7b22 krb5kdc[8520](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 01:10:28 dist-test-slave-7b22 krb5kdc[8520](info): set up 2 sockets
Sep 26 01:10:28 dist-test-slave-7b22 krb5kdc[8520](info): commencing operation
krb5kdc: starting...
W20260926 01:10:29.154078 392 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790384993574762-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790384993574762-392-0/krb5kdc/test-user.keytab.
Sep 26 01:10:29 dist-test-slave-7b22 krb5kdc[8520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385029, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T01:10:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:10:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790384993574762-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790384993574762-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 01:10:29.270969 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790384993574762-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790384993574762-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790384993574762-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46773
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790384993574762-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46773
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:10:29.350062 8533 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:10:29.350281 8533 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:10:29.350302 8533 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:10:29.351429 8533 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:10:29.351464 8533 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:10:29.351526 8533 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:10:29.351545 8533 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:10:29.351557 8533 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:10:29.353044 8533 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790384993574762-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790384993574762-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46773
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790384993574762-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46773
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8533
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790384993574762-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:10:29.353235 8533 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:10:29.353403 8533 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:10:29.358080 8539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:10:29.358074 8541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:10:29.358074 8538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:10:29.358359 8533 server_base.cc:1034] running on GCE node
I20260926 01:10:29.358588 8533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:10:29.358857 8533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:10:29.359998 8533 hybrid_clock.cc:648] HybridClock initialized: now 1790385029359970 us; error 41 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-26T01:10:29Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1172 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 01:10:29 dist-test-slave-7b22 krb5kdc[8546](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 01:10:29 dist-test-slave-7b22 krb5kdc[8546](info): set up 2 sockets
Sep 26 01:10:29 dist-test-slave-7b22 krb5kdc[8546](info): commencing operation
krb5kdc: starting...
W20260926 01:10:30.322171 392 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.004s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/krb5kdc/test-user.keytab.
Sep 26 01:10:30 dist-test-slave-7b22 krb5kdc[8546](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385030, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T01:10:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:10:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 01:10:30.435185 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32971
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:32971
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:10:30.514515 8559 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:10:30.514722 8559 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:10:30.514751 8559 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:10:30.515887 8559 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:10:30.515930 8559 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:10:30.515949 8559 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:10:30.515965 8559 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:10:30.515991 8559 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:10:30.517496 8559 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:32971
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32971
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8559
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:10:30.517732 8559 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:10:30.517969 8559 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:10:30.522497 8565 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:10:30.522517 8564 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:10:30.522584 8559 server_base.cc:1034] running on GCE node
W20260926 01:10:30.522504 8567 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:10:30.522887 8559 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:10:30.523169 8559 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:10:30.524317 8559 hybrid_clock.cc:648] HybridClock initialized: now 1790385030524302 us; error 30 us; skew 500 ppm
Sep 26 01:10:30 dist-test-slave-7b22 krb5kdc[8546](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385030, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:10:30.526624 8559 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 01:10:30.527063 8559 webserver.cc:460] Webserver started at http://127.0.98.62:35863/ using document root <none> and password file <none>
I20260926 01:10:30.527272 8559 fs_manager.cc:359] Metadata directory not provided
I20260926 01:10:30.527320 8559 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:10:30.527442 8559 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:10:30.528468 8559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/minicluster-data/master-0/data/instance:
uuid: "4b0a6fa67bca47bcaee2dbc769ef78b3"
format_stamp: "Formatted at 2026-09-26 01:10:30 on dist-test-slave-7b22"
server_key: "4227e38985ef55e8c7f7b37da6af0274"
server_key_iv: "64e1877508f7596f7834a644eb021766"
server_key_version: "encryptionkey@0"
I20260926 01:10:30.528810 8559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/minicluster-data/master-0/wal/instance:
uuid: "4b0a6fa67bca47bcaee2dbc769ef78b3"
format_stamp: "Formatted at 2026-09-26 01:10:30 on dist-test-slave-7b22"
server_key: "4227e38985ef55e8c7f7b37da6af0274"
server_key_iv: "64e1877508f7596f7834a644eb021766"
server_key_version: "encryptionkey@0"
I20260926 01:10:30.530071 8559 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 01:10:30.530751 8574 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:30.530926 8559 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:30.531008 8559 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/minicluster-data/master-0/wal
uuid: "4b0a6fa67bca47bcaee2dbc769ef78b3"
format_stamp: "Formatted at 2026-09-26 01:10:30 on dist-test-slave-7b22"
server_key: "4227e38985ef55e8c7f7b37da6af0274"
server_key_iv: "64e1877508f7596f7834a644eb021766"
server_key_version: "encryptionkey@0"
I20260926 01:10:30.531068 8559 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:10:30.552934 8559 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:10:30.555063 8559 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:10:30.555197 8559 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:10:30.559463 8559 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:32971
I20260926 01:10:30.559494 8626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:32971 every 8 connection(s)
I20260926 01:10:30.560060 8559 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
I20260926 01:10:30.560990 8627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:10:30.563879 8627 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b0a6fa67bca47bcaee2dbc769ef78b3: Bootstrap starting.
I20260926 01:10:30.564581 8627 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4b0a6fa67bca47bcaee2dbc769ef78b3: Neither blocks nor log segments found. Creating new log.
I20260926 01:10:30.564894 8627 log.cc:824] T 00000000000000000000000000000000 P 4b0a6fa67bca47bcaee2dbc769ef78b3: Log is configured to *not* fsync() on all Append() calls
I20260926 01:10:30.565521 8627 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b0a6fa67bca47bcaee2dbc769ef78b3: No bootstrap required, opened a new log
I20260926 01:10:30.567162 8627 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b0a6fa67bca47bcaee2dbc769ef78b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b0a6fa67bca47bcaee2dbc769ef78b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32971 } }
I20260926 01:10:30.567304 8627 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4b0a6fa67bca47bcaee2dbc769ef78b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:10:30.567327 8627 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b0a6fa67bca47bcaee2dbc769ef78b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b0a6fa67bca47bcaee2dbc769ef78b3, State: Initialized, Role: FOLLOWER
I20260926 01:10:30.567472 8627 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b0a6fa67bca47bcaee2dbc769ef78b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b0a6fa67bca47bcaee2dbc769ef78b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32971 } }
I20260926 01:10:30.567543 8627 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4b0a6fa67bca47bcaee2dbc769ef78b3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:10:30.567565 8627 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b0a6fa67bca47bcaee2dbc769ef78b3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:10:30.567610 8627 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b0a6fa67bca47bcaee2dbc769ef78b3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:10:30.568481 8627 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b0a6fa67bca47bcaee2dbc769ef78b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b0a6fa67bca47bcaee2dbc769ef78b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32971 } }
I20260926 01:10:30.568594 8627 leader_election.cc:302] T 00000000000000000000000000000000 P 4b0a6fa67bca47bcaee2dbc769ef78b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b0a6fa67bca47bcaee2dbc769ef78b3; no voters:
I20260926 01:10:30.568758 8627 leader_election.cc:288] T 00000000000000000000000000000000 P 4b0a6fa67bca47bcaee2dbc769ef78b3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:10:30.568807 8630 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b0a6fa67bca47bcaee2dbc769ef78b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:10:30.568981 8630 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4b0a6fa67bca47bcaee2dbc769ef78b3 [term 1 LEADER]: Becoming Leader. State: Replica: 4b0a6fa67bca47bcaee2dbc769ef78b3, State: Running, Role: LEADER
I20260926 01:10:30.568990 8627 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b0a6fa67bca47bcaee2dbc769ef78b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:10:30.569140 8630 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b0a6fa67bca47bcaee2dbc769ef78b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b0a6fa67bca47bcaee2dbc769ef78b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32971 } }
I20260926 01:10:30.569492 8632 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b0a6fa67bca47bcaee2dbc769ef78b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b0a6fa67bca47bcaee2dbc769ef78b3. Latest consensus state: current_term: 1 leader_uuid: "4b0a6fa67bca47bcaee2dbc769ef78b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b0a6fa67bca47bcaee2dbc769ef78b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32971 } } }
I20260926 01:10:30.569578 8632 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b0a6fa67bca47bcaee2dbc769ef78b3 [sys.catalog]: This master's current role is: LEADER
I20260926 01:10:30.569871 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 8559
I20260926 01:10:30.569984 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/minicluster-data/master-0/wal/instance
I20260926 01:10:30.570013 8631 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b0a6fa67bca47bcaee2dbc769ef78b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4b0a6fa67bca47bcaee2dbc769ef78b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b0a6fa67bca47bcaee2dbc769ef78b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32971 } } }
I20260926 01:10:30.570075 8638 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:10:30.570159 392 external_mini_cluster.cc:1428] Setting key 680dc9a3afc57fc2eddd99578c85285e
I20260926 01:10:30.570595 8638 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:10:30.570075 8631 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b0a6fa67bca47bcaee2dbc769ef78b3 [sys.catalog]: This master's current role is: LEADER
I20260926 01:10:30.572162 8638 catalog_manager.cc:1347] Generated new cluster ID: 50a742ffe81a4cfb8f994e0e487a3915
I20260926 01:10:30.572206 8638 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 01:10:30 dist-test-slave-7b22 krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385030, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:10:30.584672 8648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:30.570598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55984 (local address 127.0.98.62:32971)
0926 01:10:30.574002 (+ 3404us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:30.574012 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:30.574053 (+ 41us) server_negotiation.cc:407] Connection header received
0926 01:10:30.574175 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:30.574182 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:30.574239 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:30.574402 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:30.575324 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:30.576712 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:30.577765 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:30.578438 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:30.578540 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:30.581209 (+ 2669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:10:30.581213 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:10:30.581213 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:10:30.581226 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:10:30.583127 (+ 1901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:30.583487 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:30.583490 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:30.583491 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:30.583599 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:30.583769 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:30.583770 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:30.583770 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:30.584078 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:10:30.584226 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:30.584295 (+ 69us) server_negotiation.cc:299] Negotiation successful
0926 01:10:30.584431 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3357,"thread_start_us":64,"threads_started":1}
I20260926 01:10:30.588212 8638 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:10:30.588742 8638 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:10:30.591033 8638 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4b0a6fa67bca47bcaee2dbc769ef78b3: Generated new TSK 0
I20260926 01:10:30.591188 8638 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 01:10:30.618724 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32971
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:10:30.698014 8653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:10:30.698216 8653 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:10:30.698237 8653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:10:30.699460 8653 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:10:30.699508 8653 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:10:30.699534 8653 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 01:10:30.701061 8653 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32971
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8653
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:10:30.701292 8653 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:10:30.701532 8653 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:10:30.706310 8658 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:10:30.706524 8659 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:10:30.706550 8653 server_base.cc:1034] running on GCE node
W20260926 01:10:30.706547 8661 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:10:30.706856 8653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:10:30.707084 8653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:10:30.708248 8653 hybrid_clock.cc:648] HybridClock initialized: now 1790385030708234 us; error 29 us; skew 500 ppm
Sep 26 01:10:30 dist-test-slave-7b22 krb5kdc[8546](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385030, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:10:30.710994 8653 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 01:10:30.711439 8653 webserver.cc:460] Webserver started at http://127.0.98.1:43693/ using document root <none> and password file <none>
I20260926 01:10:30.711648 8653 fs_manager.cc:359] Metadata directory not provided
I20260926 01:10:30.711691 8653 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:10:30.711809 8653 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:10:30.712818 8653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/minicluster-data/ts-0/data/instance:
uuid: "46ecfa67f3db47cab2141e920871aec9"
format_stamp: "Formatted at 2026-09-26 01:10:30 on dist-test-slave-7b22"
server_key: "c50e78cbcc2e6de9e47afe204883d905"
server_key_iv: "a2e3c5cd600ca60399df7ffa16d5301e"
server_key_version: "encryptionkey@0"
I20260926 01:10:30.713126 8653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/minicluster-data/ts-0/wal/instance:
uuid: "46ecfa67f3db47cab2141e920871aec9"
format_stamp: "Formatted at 2026-09-26 01:10:30 on dist-test-slave-7b22"
server_key: "c50e78cbcc2e6de9e47afe204883d905"
server_key_iv: "a2e3c5cd600ca60399df7ffa16d5301e"
server_key_version: "encryptionkey@0"
I20260926 01:10:30.714371 8653 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 01:10:30.715044 8668 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:30.715188 8653 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 01:10:30.715268 8653 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/minicluster-data/ts-0/wal
uuid: "46ecfa67f3db47cab2141e920871aec9"
format_stamp: "Formatted at 2026-09-26 01:10:30 on dist-test-slave-7b22"
server_key: "c50e78cbcc2e6de9e47afe204883d905"
server_key_iv: "a2e3c5cd600ca60399df7ffa16d5301e"
server_key_version: "encryptionkey@0"
I20260926 01:10:30.715332 8653 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:10:30.729105 8653 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:10:30.731052 8653 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:10:30.731174 8653 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:10:30.731398 8653 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:10:30.731688 8653 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:10:30.731721 8653 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:30.731750 8653 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:10:30.731766 8653 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:30.737169 8653 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42111
I20260926 01:10:30.737210 8781 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42111 every 8 connection(s)
I20260926 01:10:30.737516 8653 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/minicluster-data/ts-0/data/info.pb
I20260926 01:10:30.743556 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 8653
I20260926 01:10:30.743652 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/minicluster-data/ts-0/wal/instance
Sep 26 01:10:30 dist-test-slave-7b22 krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385030, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:10:30.743813 392 external_mini_cluster.cc:1428] Setting key ef2452e1e60447c3ce50d40a62a9f32f
I20260926 01:10:30.747084 8648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:30.738245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34739 (local address 127.0.98.62:32971)
0926 01:10:30.738303 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:30.738307 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:30.738518 (+ 211us) server_negotiation.cc:407] Connection header received
0926 01:10:30.739438 (+ 920us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:30.739439 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:30.739450 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:30.739502 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:30.740155 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:30.740685 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:30.741684 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:30.742003 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:30.742069 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:30.744241 (+ 2172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:10:30.744242 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:10:30.744243 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:10:30.744250 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:10:30.745700 (+ 1450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:30.746199 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:30.746200 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:30.746201 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:30.746259 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:30.746487 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:30.746488 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:30.746489 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:30.746599 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:10:30.746657 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:30.746944 (+ 287us) server_negotiation.cc:299] Negotiation successful
0926 01:10:30.747017 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 01:10:30.747330 8784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:30.738230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32971 (local address 127.0.98.1:34739)
0926 01:10:30.738404 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:30.738439 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:30.739343 (+ 904us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:30.739567 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:30.739571 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:30.739831 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:10:30.740045 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:30.740049 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:30.740767 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:30.740768 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:30.741593 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:30.741596 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:30.742097 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:30.742099 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:30.742216 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:30.742694 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:10:30.742702 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:10:30.744172 (+ 1470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:10:30.745793 (+ 1621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:10:30.745800 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:10:30.745802 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:10:30.746116 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:10:30.746316 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:10:30.746317 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:10:30.746317 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:10:30.746428 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:10:30.746655 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:10:30.746661 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:10:30.746806 (+ 145us) client_negotiation.cc:769] Sending connection context
0926 01:10:30.746922 (+ 116us) client_negotiation.cc:240] Negotiation successful
0926 01:10:30.747091 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":95,"threads_started":1}
I20260926 01:10:30.747855 8782 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32971
I20260926 01:10:30.747956 8782 heartbeater.cc:464] Registering TS with master...
I20260926 01:10:30.748150 8782 heartbeater.cc:511] Master 127.0.98.62:32971 requested a full tablet report, sending...
I20260926 01:10:30.748670 8591 ts_manager.cc:194] Registered new tserver with Master: 46ecfa67f3db47cab2141e920871aec9 (127.0.98.1:42111)
I20260926 01:10:30.749272 8591 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:34739
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 01:10:30.768703 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32971
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:10:30.848058 8787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:10:30.848263 8787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:10:30.848304 8787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:10:30.849506 8787 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:10:30.849545 8787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:10:30.849579 8787 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 01:10:30.851112 8787 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32971
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8787
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:10:30.851356 8787 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:10:30.851554 8787 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:10:30.856473 8793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:10:30.856454 8792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:10:30.856516 8795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:10:30.856633 8787 server_base.cc:1034] running on GCE node
I20260926 01:10:30.856875 8787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:10:30.857074 8787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:10:30.858233 8787 hybrid_clock.cc:648] HybridClock initialized: now 1790385030858214 us; error 34 us; skew 500 ppm
Sep 26 01:10:30 dist-test-slave-7b22 krb5kdc[8546](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385030, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:10:30.860623 8787 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 01:10:30.861065 8787 webserver.cc:460] Webserver started at http://127.0.98.2:41603/ using document root <none> and password file <none>
I20260926 01:10:30.861269 8787 fs_manager.cc:359] Metadata directory not provided
I20260926 01:10:30.861315 8787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:10:30.861438 8787 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:10:30.862550 8787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/minicluster-data/ts-1/data/instance:
uuid: "3ce481c401a74ab0821c12a25e244480"
format_stamp: "Formatted at 2026-09-26 01:10:30 on dist-test-slave-7b22"
server_key: "55bae24772a49d7a02db6ae479bd5f31"
server_key_iv: "74ce5497f83f98f0d55c3343ccfd97ff"
server_key_version: "encryptionkey@0"
I20260926 01:10:30.862874 8787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/minicluster-data/ts-1/wal/instance:
uuid: "3ce481c401a74ab0821c12a25e244480"
format_stamp: "Formatted at 2026-09-26 01:10:30 on dist-test-slave-7b22"
server_key: "55bae24772a49d7a02db6ae479bd5f31"
server_key_iv: "74ce5497f83f98f0d55c3343ccfd97ff"
server_key_version: "encryptionkey@0"
I20260926 01:10:30.864207 8787 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 01:10:30.865051 8802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:30.865201 8787 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 01:10:30.865283 8787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/minicluster-data/ts-1/wal
uuid: "3ce481c401a74ab0821c12a25e244480"
format_stamp: "Formatted at 2026-09-26 01:10:30 on dist-test-slave-7b22"
server_key: "55bae24772a49d7a02db6ae479bd5f31"
server_key_iv: "74ce5497f83f98f0d55c3343ccfd97ff"
server_key_version: "encryptionkey@0"
I20260926 01:10:30.865370 8787 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:10:30.899708 8787 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:10:30.901813 8787 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:10:30.901980 8787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:10:30.902211 8787 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:10:30.902518 8787 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:10:30.902550 8787 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:30.902572 8787 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:10:30.902582 8787 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:30.908071 8787 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40353
I20260926 01:10:30.908116 8915 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40353 every 8 connection(s)
I20260926 01:10:30.908418 8787 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/minicluster-data/ts-1/data/info.pb
I20260926 01:10:30.913743 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 8787
I20260926 01:10:30.913919 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/minicluster-data/ts-1/wal/instance
I20260926 01:10:30.914116 392 external_mini_cluster.cc:1428] Setting key 7f90c86d588eb75028f140ce5397751b
Sep 26 01:10:30 dist-test-slave-7b22 krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385030, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:10:30.917614 8648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:30.909118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36325 (local address 127.0.98.62:32971)
0926 01:10:30.909178 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:30.909180 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:30.909384 (+ 204us) server_negotiation.cc:407] Connection header received
0926 01:10:30.910322 (+ 938us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:30.910324 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:30.910334 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:30.910404 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:30.910976 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:30.911465 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:30.912488 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:30.912710 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:30.912760 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:30.915079 (+ 2319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:10:30.915080 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:10:30.915081 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:10:30.915088 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:10:30.916376 (+ 1288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:30.916730 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:30.916731 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:30.916731 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:30.916794 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:30.917014 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:30.917015 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:30.917015 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:30.917127 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:10:30.917182 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:30.917442 (+ 260us) server_negotiation.cc:299] Negotiation successful
0926 01:10:30.917525 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 01:10:30.917783 8918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:30.909090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32971 (local address 127.0.98.2:36325)
0926 01:10:30.909238 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:30.909281 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:30.910218 (+ 937us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:30.910461 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:30.910464 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:30.910711 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:10:30.910905 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:30.910909 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:30.911561 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:30.911562 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:30.912418 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:30.912421 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:30.912771 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:30.912772 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:30.912916 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:30.913425 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:10:30.913432 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:10:30.914999 (+ 1567us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:10:30.916449 (+ 1450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:10:30.916455 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:10:30.916458 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:10:30.916665 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:10:30.916848 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:10:30.916849 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:10:30.916849 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:10:30.916959 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:10:30.917182 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:10:30.917189 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:10:30.917329 (+ 140us) client_negotiation.cc:769] Sending connection context
0926 01:10:30.917417 (+ 88us) client_negotiation.cc:240] Negotiation successful
0926 01:10:30.917572 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":67,"threads_started":1}
I20260926 01:10:30.918413 8916 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32971
I20260926 01:10:30.918502 8916 heartbeater.cc:464] Registering TS with master...
I20260926 01:10:30.918686 8916 heartbeater.cc:511] Master 127.0.98.62:32971 requested a full tablet report, sending...
I20260926 01:10:30.919063 8591 ts_manager.cc:194] Registered new tserver with Master: 3ce481c401a74ab0821c12a25e244480 (127.0.98.2:40353)
I20260926 01:10:30.919458 8591 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:36325
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 01:10:30.940277 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32971
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:10:31.019184 8921 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:10:31.019404 8921 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:10:31.019436 8921 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:10:31.020646 8921 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:10:31.020699 8921 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:10:31.020735 8921 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 01:10:31.022307 8921 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32971
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8921
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:10:31.022543 8921 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:10:31.022755 8921 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:10:31.027521 8927 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:10:31.027609 8926 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:10:31.027628 8929 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:10:31.027765 8921 server_base.cc:1034] running on GCE node
I20260926 01:10:31.027966 8921 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:10:31.028174 8921 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:10:31.029301 8921 hybrid_clock.cc:648] HybridClock initialized: now 1790385031029288 us; error 28 us; skew 500 ppm
Sep 26 01:10:31 dist-test-slave-7b22 krb5kdc[8546](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385031, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:10:31.031836 8921 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 01:10:31.032258 8921 webserver.cc:460] Webserver started at http://127.0.98.3:39251/ using document root <none> and password file <none>
I20260926 01:10:31.032473 8921 fs_manager.cc:359] Metadata directory not provided
I20260926 01:10:31.032522 8921 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:10:31.032645 8921 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:10:31.033780 8921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/minicluster-data/ts-2/data/instance:
uuid: "6450c7a9fd524f70826c30488374d0c6"
format_stamp: "Formatted at 2026-09-26 01:10:31 on dist-test-slave-7b22"
server_key: "ad15ffd284a51111bc392859580c7291"
server_key_iv: "0e7f2f459d3026847f7817c3651292a8"
server_key_version: "encryptionkey@0"
I20260926 01:10:31.034168 8921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/minicluster-data/ts-2/wal/instance:
uuid: "6450c7a9fd524f70826c30488374d0c6"
format_stamp: "Formatted at 2026-09-26 01:10:31 on dist-test-slave-7b22"
server_key: "ad15ffd284a51111bc392859580c7291"
server_key_iv: "0e7f2f459d3026847f7817c3651292a8"
server_key_version: "encryptionkey@0"
I20260926 01:10:31.035513 8921 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:10:31.036269 8936 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:31.036454 8921 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:10:31.036540 8921 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/minicluster-data/ts-2/wal
uuid: "6450c7a9fd524f70826c30488374d0c6"
format_stamp: "Formatted at 2026-09-26 01:10:31 on dist-test-slave-7b22"
server_key: "ad15ffd284a51111bc392859580c7291"
server_key_iv: "0e7f2f459d3026847f7817c3651292a8"
server_key_version: "encryptionkey@0"
I20260926 01:10:31.036595 8921 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:10:31.046452 8921 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:10:31.048435 8921 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:10:31.048561 8921 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:10:31.048746 8921 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:10:31.049088 8921 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:10:31.049124 8921 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:31.049152 8921 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:10:31.049186 8921 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:31.054569 8921 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45833
I20260926 01:10:31.054632 9049 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45833 every 8 connection(s)
I20260926 01:10:31.054919 8921 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/minicluster-data/ts-2/data/info.pb
Sep 26 01:10:31 dist-test-slave-7b22 krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385031, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:10:31.064075 8648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:31.055598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54355 (local address 127.0.98.62:32971)
0926 01:10:31.055677 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:31.055679 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:31.055881 (+ 202us) server_negotiation.cc:407] Connection header received
0926 01:10:31.056810 (+ 929us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:31.056811 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:31.056822 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:31.056875 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:31.057462 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:31.058029 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:31.058980 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:31.059181 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:31.059237 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:31.061484 (+ 2247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:10:31.061486 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:10:31.061486 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:10:31.061492 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:10:31.062787 (+ 1295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:31.063187 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:31.063188 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:31.063188 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:31.063236 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:31.063483 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:31.063485 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:31.063485 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:31.063608 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:10:31.063671 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:31.063938 (+ 267us) server_negotiation.cc:299] Negotiation successful
0926 01:10:31.064004 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 01:10:31.064268 9052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:31.055544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32971 (local address 127.0.98.3:54355)
0926 01:10:31.055715 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:31.055751 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:31.056713 (+ 962us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:31.056946 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:31.056949 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:31.057193 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:10:31.057379 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:31.057383 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:31.058111 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:31.058112 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:31.058902 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:31.058904 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:31.059245 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:31.059246 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:31.059361 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:31.059897 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:10:31.059904 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:10:31.061361 (+ 1457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:10:31.062877 (+ 1516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:10:31.062884 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:10:31.062886 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:10:31.063118 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:10:31.063296 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:10:31.063296 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:10:31.063297 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:10:31.063399 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:10:31.063691 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:10:31.063697 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:10:31.063823 (+ 126us) client_negotiation.cc:769] Sending connection context
0926 01:10:31.063921 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 01:10:31.064046 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":76,"threads_started":1}
I20260926 01:10:31.064914 9050 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32971
I20260926 01:10:31.065003 9050 heartbeater.cc:464] Registering TS with master...
I20260926 01:10:31.064981 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 8921
I20260926 01:10:31.065078 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/minicluster-data/ts-2/wal/instance
I20260926 01:10:31.065182 9050 heartbeater.cc:511] Master 127.0.98.62:32971 requested a full tablet report, sending...
I20260926 01:10:31.065246 392 external_mini_cluster.cc:1428] Setting key 873fd5f8ae8f3b3b96130273722658bb
I20260926 01:10:31.065536 8591 ts_manager.cc:194] Registered new tserver with Master: 6450c7a9fd524f70826c30488374d0c6 (127.0.98.3:45833)
I20260926 01:10:31.066035 8591 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:54355
I20260926 01:10:31.066143 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/krb5kdc/oryx_127.0.98.1.keytab.
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/krb5kdc/oryx_127.0.98.2.keytab.
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260926 01:10:31.138697 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 8653
I20260926 01:10:31.142690 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 8787
I20260926 01:10:31.146454 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 8921
I20260926 01:10:31.150322 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 8559
I20260926 01:10:31.154748 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32971
--webserver_interface=127.0.98.62
--webserver_port=35863
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:32971
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:10:31.233616 9059 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:10:31.233858 9059 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:10:31.233881 9059 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:10:31.235051 9059 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:10:31.235093 9059 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:10:31.235106 9059 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:10:31.235117 9059 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:10:31.235129 9059 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:10:31.236663 9059 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:32971
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32971
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=35863
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9059
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:10:31.236898 9059 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:10:31.237115 9059 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:10:31.241802 9065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:10:31.241891 9064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:10:31.242156 9059 server_base.cc:1034] running on GCE node
W20260926 01:10:31.241803 9067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:10:31.242501 9059 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:10:31.242815 9059 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:10:31.243984 9059 hybrid_clock.cc:648] HybridClock initialized: now 1790385031243961 us; error 28 us; skew 500 ppm
Sep 26 01:10:31 dist-test-slave-7b22 krb5kdc[8546](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385031, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:10:31.246428 9059 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 01:10:31.246883 9059 webserver.cc:460] Webserver started at http://127.0.98.62:35863/ using document root <none> and password file <none>
I20260926 01:10:31.247103 9059 fs_manager.cc:359] Metadata directory not provided
I20260926 01:10:31.247165 9059 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:10:31.248472 9059 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:10:31.249125 9074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:31.249272 9059 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 01:10:31.249374 9059 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/minicluster-data/master-0/wal
uuid: "4b0a6fa67bca47bcaee2dbc769ef78b3"
format_stamp: "Formatted at 2026-09-26 01:10:30 on dist-test-slave-7b22"
server_key: "4227e38985ef55e8c7f7b37da6af0274"
server_key_iv: "64e1877508f7596f7834a644eb021766"
server_key_version: "encryptionkey@0"
I20260926 01:10:31.249729 9059 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:10:31.265765 9059 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:10:31.268164 9059 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:10:31.268283 9059 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:10:31.272070 9059 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:32971
I20260926 01:10:31.272115 9126 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:32971 every 8 connection(s)
I20260926 01:10:31.272601 9059 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
I20260926 01:10:31.273901 9127 sys_catalog.cc:263] Verifying existing consensus state
I20260926 01:10:31.274537 9127 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b0a6fa67bca47bcaee2dbc769ef78b3: Bootstrap starting.
I20260926 01:10:31.279249 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 9059
I20260926 01:10:31.279533 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:42111
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:32971
--webserver_port=43693
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.1
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/krb5kdc/oryx_127.0.98.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 01:10:31.282675 9127 log.cc:824] T 00000000000000000000000000000000 P 4b0a6fa67bca47bcaee2dbc769ef78b3: Log is configured to *not* fsync() on all Append() calls
I20260926 01:10:31.284264 9127 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b0a6fa67bca47bcaee2dbc769ef78b3: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 01:10:31.284528 9127 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b0a6fa67bca47bcaee2dbc769ef78b3: Bootstrap complete.
I20260926 01:10:31.286736 9127 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b0a6fa67bca47bcaee2dbc769ef78b3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b0a6fa67bca47bcaee2dbc769ef78b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32971 } }
I20260926 01:10:31.286970 9127 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b0a6fa67bca47bcaee2dbc769ef78b3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b0a6fa67bca47bcaee2dbc769ef78b3, State: Initialized, Role: FOLLOWER
I20260926 01:10:31.287123 9127 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b0a6fa67bca47bcaee2dbc769ef78b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b0a6fa67bca47bcaee2dbc769ef78b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32971 } }
I20260926 01:10:31.287197 9127 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4b0a6fa67bca47bcaee2dbc769ef78b3 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:10:31.287245 9127 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b0a6fa67bca47bcaee2dbc769ef78b3 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:10:31.287290 9127 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b0a6fa67bca47bcaee2dbc769ef78b3 [term 1 FOLLOWER]: Advancing to term 2
I20260926 01:10:31.288298 9127 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b0a6fa67bca47bcaee2dbc769ef78b3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b0a6fa67bca47bcaee2dbc769ef78b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32971 } }
I20260926 01:10:31.288403 9127 leader_election.cc:302] T 00000000000000000000000000000000 P 4b0a6fa67bca47bcaee2dbc769ef78b3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b0a6fa67bca47bcaee2dbc769ef78b3; no voters:
I20260926 01:10:31.288579 9127 leader_election.cc:288] T 00000000000000000000000000000000 P 4b0a6fa67bca47bcaee2dbc769ef78b3 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 01:10:31.288655 9131 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b0a6fa67bca47bcaee2dbc769ef78b3 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 01:10:31.288761 9131 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4b0a6fa67bca47bcaee2dbc769ef78b3 [term 2 LEADER]: Becoming Leader. State: Replica: 4b0a6fa67bca47bcaee2dbc769ef78b3, State: Running, Role: LEADER
I20260926 01:10:31.288849 9127 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b0a6fa67bca47bcaee2dbc769ef78b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:10:31.288842 9131 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b0a6fa67bca47bcaee2dbc769ef78b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b0a6fa67bca47bcaee2dbc769ef78b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32971 } }
I20260926 01:10:31.289146 9131 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b0a6fa67bca47bcaee2dbc769ef78b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b0a6fa67bca47bcaee2dbc769ef78b3. Latest consensus state: current_term: 2 leader_uuid: "4b0a6fa67bca47bcaee2dbc769ef78b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b0a6fa67bca47bcaee2dbc769ef78b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32971 } } }
I20260926 01:10:31.289280 9131 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b0a6fa67bca47bcaee2dbc769ef78b3 [sys.catalog]: This master's current role is: LEADER
I20260926 01:10:31.289475 9132 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b0a6fa67bca47bcaee2dbc769ef78b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "4b0a6fa67bca47bcaee2dbc769ef78b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b0a6fa67bca47bcaee2dbc769ef78b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32971 } } }
I20260926 01:10:31.289575 9132 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b0a6fa67bca47bcaee2dbc769ef78b3 [sys.catalog]: This master's current role is: LEADER
I20260926 01:10:31.290555 9147 catalog_manager.cc:1259] Loaded cluster ID: 50a742ffe81a4cfb8f994e0e487a3915
I20260926 01:10:31.290606 9147 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4b0a6fa67bca47bcaee2dbc769ef78b3: loading cluster ID for follower catalog manager: success
I20260926 01:10:31.291723 9147 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4b0a6fa67bca47bcaee2dbc769ef78b3: acquiring CA information for follower catalog manager: success
I20260926 01:10:31.292071 9147 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4b0a6fa67bca47bcaee2dbc769ef78b3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 01:10:31.297451 9139 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:10:31.297683 9139 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:10:31.297750 9139 catalog_manager.cc:1259] Loaded cluster ID: 50a742ffe81a4cfb8f994e0e487a3915
I20260926 01:10:31.297778 9139 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 01:10:31.298019 9139 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:10:31.298107 9139 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4b0a6fa67bca47bcaee2dbc769ef78b3: Loaded TSK: 0
I20260926 01:10:31.298262 9139 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 01:10:31.361689 9129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:10:31.361977 9129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:10:31.362017 9129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:10:31.363301 9129 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:10:31.363349 9129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:10:31.363387 9129 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 01:10:31.364879 9129 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:42111
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=43693
--tserver_master_addrs=127.0.98.62:32971
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9129
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:10:31.365123 9129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:10:31.365342 9129 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:10:31.370276 9153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:10:31.370406 9156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:10:31.370289 9154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:10:31.370621 9129 server_base.cc:1034] running on GCE node
I20260926 01:10:31.370878 9129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:10:31.371114 9129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:10:31.372262 9129 hybrid_clock.cc:648] HybridClock initialized: now 1790385031372246 us; error 34 us; skew 500 ppm
Sep 26 01:10:31 dist-test-slave-7b22 krb5kdc[8546](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385031, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:10:31.379690 9129 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260926 01:10:31.380309 9129 webserver.cc:460] Webserver started at http://127.0.98.1:43693/ using document root <none> and password file <none>
I20260926 01:10:31.380529 9129 fs_manager.cc:359] Metadata directory not provided
I20260926 01:10:31.380597 9129 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:10:31.382157 9129 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:10:31.382917 9163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:31.383106 9129 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:10:31.383222 9129 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/minicluster-data/ts-0/wal
uuid: "46ecfa67f3db47cab2141e920871aec9"
format_stamp: "Formatted at 2026-09-26 01:10:30 on dist-test-slave-7b22"
server_key: "c50e78cbcc2e6de9e47afe204883d905"
server_key_iv: "a2e3c5cd600ca60399df7ffa16d5301e"
server_key_version: "encryptionkey@0"
I20260926 01:10:31.383633 9129 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:10:31.399718 9129 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:10:31.402233 9129 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:10:31.402380 9129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:10:31.402611 9129 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:10:31.402940 9129 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:10:31.402980 9129 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:31.403009 9129 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:10:31.403033 9129 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:31.408871 9129 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42111
I20260926 01:10:31.408936 9276 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42111 every 8 connection(s)
I20260926 01:10:31.409260 9129 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/minicluster-data/ts-0/data/info.pb
I20260926 01:10:31.414883 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 9129
I20260926 01:10:31.415215 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:40353
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:32971
--webserver_port=41603
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.2
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/krb5kdc/oryx_127.0.98.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 01:10:31 dist-test-slave-7b22 krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:10:31 dist-test-slave-7b22 krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:10:31.419301 9279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:31.410023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32971 (local address 127.0.98.1:45729)
0926 01:10:31.410204 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:31.410252 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:31.410922 (+ 670us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:31.411585 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:31.411589 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:31.412220 (+ 631us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:10:31.412412 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:31.412417 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:31.414069 (+ 1652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:31.414070 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:31.415177 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:31.415179 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:31.415994 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:31.415996 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:31.416150 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:31.416714 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:10:31.416723 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:10:31.419100 (+ 2377us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:32971: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":89,"threads_started":1}
W20260926 01:10:31.419466 9280 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:10:31.410117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45729 (local address 127.0.98.62:32971)
0926 01:10:31.411133 (+ 1016us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:31.411143 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:31.411188 (+ 45us) server_negotiation.cc:407] Connection header received
0926 01:10:31.411326 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:31.411333 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:31.411392 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:31.411524 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:31.412528 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:31.413972 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:31.415265 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:31.415894 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:31.415995 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:31.419280 (+ 3285us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:45729: BlockingRecv error: recv got EOF from 127.0.98.1:45729 (error 108)
Metrics: {"server-negotiator.queue_time_us":960,"thread_start_us":77,"threads_started":1}
W20260926 01:10:31.419943 9277 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:32971 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:32971: Client connection negotiation failed: client connection to 127.0.98.62:32971: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 26 01:10:31 dist-test-slave-7b22 krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:10:31 dist-test-slave-7b22 krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:10:31.424885 9279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:31.420170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32971 (local address 127.0.98.1:48071)
0926 01:10:31.420214 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:31.420223 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:31.420257 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:31.420395 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:31.420395 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:31.420539 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:10:31.420610 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:31.420612 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:31.421519 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:31.421520 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:31.422159 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:31.422162 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:31.422598 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:31.422599 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:31.422661 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:31.423101 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:10:31.423106 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:10:31.424815 (+ 1709us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:32971: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":29}
W20260926 01:10:31.424995 9280 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:10:31.420224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48071 (local address 127.0.98.62:32971)
0926 01:10:31.420262 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:31.420264 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:31.420276 (+ 12us) server_negotiation.cc:407] Connection header received
0926 01:10:31.420305 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:31.420306 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:31.420318 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:31.420389 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:31.420681 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:31.421434 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:31.422233 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:31.422520 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:31.422594 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:31.424929 (+ 2335us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:48071: BlockingRecv error: recv got EOF from 127.0.98.1:48071 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
Sep 26 01:10:31 dist-test-slave-7b22 krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:10:31 dist-test-slave-7b22 krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:10:31.430675 9282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:31.425131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32971 (local address 127.0.98.1:51237)
0926 01:10:31.425267 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:31.425277 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:31.425332 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:31.425451 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:31.425451 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:31.425581 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:10:31.425634 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:31.425635 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:31.426572 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:31.426572 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:31.427148 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:31.427150 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:31.427588 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:31.427589 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:31.427637 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:31.428112 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:10:31.428117 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:10:31.430604 (+ 2487us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:32971: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":80,"threads_started":1}
W20260926 01:10:31.430802 9280 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:10:31.425249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51237 (local address 127.0.98.62:32971)
0926 01:10:31.425280 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:31.425282 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:31.425332 (+ 50us) server_negotiation.cc:407] Connection header received
0926 01:10:31.425373 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:31.425374 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:31.425384 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:31.425442 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:31.425686 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:31.426493 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:31.427215 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:31.427510 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:31.427571 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:31.430733 (+ 3162us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:51237: BlockingRecv error: recv got EOF from 127.0.98.1:51237 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260926 01:10:31.430819 9277 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 01:10:31.498467 9281 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:10:31.498687 9281 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:10:31.498718 9281 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:10:31.499912 9281 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:10:31.499953 9281 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:10:31.499989 9281 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 01:10:31.501431 9281 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:40353
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=41603
--tserver_master_addrs=127.0.98.62:32971
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9281
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:10:31.501664 9281 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:10:31.501891 9281 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 01:10:31.506690 9281 server_base.cc:1034] running on GCE node
W20260926 01:10:31.506594 9288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:10:31.506593 9290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:10:31.506615 9287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:10:31.507043 9281 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:10:31.507282 9281 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:10:31.508443 9281 hybrid_clock.cc:648] HybridClock initialized: now 1790385031508426 us; error 33 us; skew 500 ppm
Sep 26 01:10:31 dist-test-slave-7b22 krb5kdc[8546](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385031, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:10:31.511046 9281 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260926 01:10:31.511516 9281 webserver.cc:460] Webserver started at http://127.0.98.2:41603/ using document root <none> and password file <none>
I20260926 01:10:31.511713 9281 fs_manager.cc:359] Metadata directory not provided
I20260926 01:10:31.511771 9281 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:10:31.513072 9281 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:10:31.513643 9297 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:31.513809 9281 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 01:10:31.513962 9281 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/minicluster-data/ts-1/wal
uuid: "3ce481c401a74ab0821c12a25e244480"
format_stamp: "Formatted at 2026-09-26 01:10:30 on dist-test-slave-7b22"
server_key: "55bae24772a49d7a02db6ae479bd5f31"
server_key_iv: "74ce5497f83f98f0d55c3343ccfd97ff"
server_key_version: "encryptionkey@0"
I20260926 01:10:31.514290 9281 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:10:31.527275 9281 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:10:31.529299 9281 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:10:31.529430 9281 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:10:31.529641 9281 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:10:31.529953 9281 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:10:31.529987 9281 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:31.530018 9281 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:10:31.530041 9281 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:31.535514 9281 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40353
I20260926 01:10:31.535622 9410 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40353 every 8 connection(s)
I20260926 01:10:31.535910 9281 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/minicluster-data/ts-1/data/info.pb
I20260926 01:10:31.540702 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 9281
I20260926 01:10:31.540963 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:45833
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:32971
--webserver_port=39251
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.3
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/krb5kdc/oryx_127.0.98.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 01:10:31 dist-test-slave-7b22 krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:10:31 dist-test-slave-7b22 krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:10:31.543205 9413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:31.536569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32971 (local address 127.0.98.2:54555)
0926 01:10:31.536702 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:31.536734 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:31.537212 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:31.537457 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:31.537460 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:31.538055 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:10:31.538244 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:31.538248 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:31.538974 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:31.538975 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:31.539772 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:31.539774 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:31.540143 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:31.540144 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:31.540259 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:31.540747 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:10:31.540757 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:10:31.543055 (+ 2298us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:32971: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":74,"threads_started":1}
W20260926 01:10:31.543219 9280 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:10:31.536566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54555 (local address 127.0.98.62:32971)
0926 01:10:31.536607 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:31.536608 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:31.536822 (+ 214us) server_negotiation.cc:407] Connection header received
0926 01:10:31.537309 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:31.537310 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:31.537325 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:31.537389 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:31.538337 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:31.538834 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:31.539874 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:31.540072 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:31.540128 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:31.543160 (+ 3032us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:54555: BlockingRecv error: recv got EOF from 127.0.98.2:54555 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260926 01:10:31.543426 9411 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:32971 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:32971: Client connection negotiation failed: client connection to 127.0.98.62:32971: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 26 01:10:31 dist-test-slave-7b22 krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:10:31 dist-test-slave-7b22 krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:10:31.548240 9413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:31.543652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32971 (local address 127.0.98.2:37961)
0926 01:10:31.543692 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:31.543702 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:31.543736 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:31.543889 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:31.543890 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:31.543994 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:10:31.544042 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:31.544044 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:31.544912 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:31.544913 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:31.545411 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:31.545413 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:31.545875 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:31.545876 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:31.545926 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:31.546344 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:10:31.546348 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:10:31.548188 (+ 1840us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:32971: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":26}
W20260926 01:10:31.548375 9280 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:10:31.543692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37961 (local address 127.0.98.62:32971)
0926 01:10:31.543728 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:31.543730 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:31.543738 (+ 8us) server_negotiation.cc:407] Connection header received
0926 01:10:31.543777 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:31.543778 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:31.543786 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:31.543841 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:31.544112 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:31.544817 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:31.545483 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:31.545770 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:31.545872 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:31.548300 (+ 2428us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:37961: BlockingRecv error: recv got EOF from 127.0.98.2:37961 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
Sep 26 01:10:31 dist-test-slave-7b22 krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:10:31 dist-test-slave-7b22 krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:10:31.552945 9413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:31.548669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32971 (local address 127.0.98.2:49869)
0926 01:10:31.548711 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:31.548721 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:31.548761 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:31.548884 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:31.548884 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:31.549006 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:10:31.549035 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:31.549036 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:31.549621 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:31.549622 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:31.550176 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:31.550178 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:31.550486 (+ 308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:31.550487 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:31.550533 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:31.551037 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:10:31.551042 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:10:31.552871 (+ 1829us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:32971: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":26}
W20260926 01:10:31.553081 9280 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:10:31.548740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49869 (local address 127.0.98.62:32971)
0926 01:10:31.548793 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:31.548795 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:31.548802 (+ 7us) server_negotiation.cc:407] Connection header received
0926 01:10:31.548824 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:31.548825 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:31.548834 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:31.548874 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:31.549096 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:31.549559 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:31.550252 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:31.550429 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:31.550482 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:31.553025 (+ 2543us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:49869: BlockingRecv error: recv got EOF from 127.0.98.2:49869 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
W20260926 01:10:31.553081 9411 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 01:10:31.623862 9414 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:10:31.624087 9414 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:10:31.624111 9414 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:10:31.625413 9414 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:10:31.625468 9414 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:10:31.625497 9414 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 01:10:31.627029 9414 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:45833
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=39251
--tserver_master_addrs=127.0.98.62:32971
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9414
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:10:31.627257 9414 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:10:31.627482 9414 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:10:31.632783 9420 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:10:31.632772 9419 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:10:31.632774 9414 server_base.cc:1034] running on GCE node
W20260926 01:10:31.632797 9422 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:10:31.633219 9414 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:10:31.633461 9414 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:10:31.634635 9414 hybrid_clock.cc:648] HybridClock initialized: now 1790385031634603 us; error 52 us; skew 500 ppm
Sep 26 01:10:31 dist-test-slave-7b22 krb5kdc[8546](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385031, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:10:31.637166 9414 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260926 01:10:31.637616 9414 webserver.cc:460] Webserver started at http://127.0.98.3:39251/ using document root <none> and password file <none>
I20260926 01:10:31.637804 9414 fs_manager.cc:359] Metadata directory not provided
I20260926 01:10:31.637899 9414 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:10:31.639235 9414 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:10:31.639803 9429 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:31.639992 9414 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 01:10:31.640065 9414 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/minicluster-data/ts-2/wal
uuid: "6450c7a9fd524f70826c30488374d0c6"
format_stamp: "Formatted at 2026-09-26 01:10:31 on dist-test-slave-7b22"
server_key: "ad15ffd284a51111bc392859580c7291"
server_key_iv: "0e7f2f459d3026847f7817c3651292a8"
server_key_version: "encryptionkey@0"
I20260926 01:10:31.640412 9414 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:10:31.649523 9414 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:10:31.651687 9414 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:10:31.651803 9414 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:10:31.652024 9414 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:10:31.652310 9414 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:10:31.652344 9414 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:31.652374 9414 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:10:31.652396 9414 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:31.658077 9414 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45833
I20260926 01:10:31.658121 9542 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45833 every 8 connection(s)
I20260926 01:10:31.658449 9414 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384993574762-392-0/minicluster-data/ts-2/data/info.pb
Sep 26 01:10:31 dist-test-slave-7b22 krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:10:31 dist-test-slave-7b22 krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:10:31.666105 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 9414
I20260926 01:10:31.666172 9545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:31.659113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32971 (local address 127.0.98.3:47727)
0926 01:10:31.659267 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:31.659308 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:31.659806 (+ 498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:31.660057 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:31.660060 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:31.660634 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:10:31.660837 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:31.660843 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:31.661589 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:31.661590 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:31.662559 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:31.662562 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:31.662954 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:31.662955 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:31.663080 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:31.663682 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:10:31.663691 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:10:31.666006 (+ 2315us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:32971: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":63,"threads_started":1}
W20260926 01:10:31.666181 9280 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:10:31.659191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47727 (local address 127.0.98.62:32971)
0926 01:10:31.659247 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:31.659249 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:31.659373 (+ 124us) server_negotiation.cc:407] Connection header received
0926 01:10:31.659920 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:31.659921 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:31.659933 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:31.660000 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:31.660961 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:31.661479 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:31.662648 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:31.662846 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:31.662909 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:31.666121 (+ 3212us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:47727: BlockingRecv error: recv got EOF from 127.0.98.3:47727 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
W20260926 01:10:31.666527 9543 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:32971 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:32971: Client connection negotiation failed: client connection to 127.0.98.62:32971: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 26 01:10:31 dist-test-slave-7b22 krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:10:31 dist-test-slave-7b22 krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:10:31.671335 9545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:31.666754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32971 (local address 127.0.98.3:50125)
0926 01:10:31.666791 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:31.666801 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:31.666851 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:31.667011 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:31.667011 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:31.667143 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:10:31.667196 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:31.667197 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:31.667856 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:31.667857 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:31.668391 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:31.668394 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:31.668764 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:31.668765 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:31.668830 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:31.669272 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:10:31.669277 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:10:31.671254 (+ 1977us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:32971: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":19}
W20260926 01:10:31.671447 9547 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:10:31.666793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50125 (local address 127.0.98.62:32971)
0926 01:10:31.666894 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:31.666896 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:31.666902 (+ 6us) server_negotiation.cc:407] Connection header received
0926 01:10:31.666925 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:31.666926 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:31.666936 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:31.667001 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:31.667276 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:31.667780 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:31.668472 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:31.668688 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:31.668752 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:31.671383 (+ 2631us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:50125: BlockingRecv error: recv got EOF from 127.0.98.3:50125 (error 108)
Metrics: {"server-negotiator.queue_time_us":86,"thread_start_us":54,"threads_started":1}
I20260926 01:10:31.673705 9280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:31.666652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35220 (local address 127.0.98.62:32971)
0926 01:10:31.666695 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:31.666697 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:31.666783 (+ 86us) server_negotiation.cc:407] Connection header received
0926 01:10:31.666886 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:31.666888 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:31.666898 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:31.666946 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:31.667498 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:31.667953 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:31.668713 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:31.668892 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:31.668944 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:31.670132 (+ 1188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:10:31.670137 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:10:31.670138 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:10:31.670152 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:10:31.672004 (+ 1852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:31.672421 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:31.672425 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:31.672425 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:31.672575 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:31.672822 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:31.672823 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:31.672823 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:31.673207 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:10:31.673370 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:31.673500 (+ 130us) server_negotiation.cc:299] Negotiation successful
0926 01:10:31.673609 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
Sep 26 01:10:31 dist-test-slave-7b22 krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:10:31 dist-test-slave-7b22 krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:10:31.677559 9545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:31.672357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32971 (local address 127.0.98.3:41841)
0926 01:10:31.672407 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:31.672416 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:31.672448 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:31.672634 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:31.672634 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:31.672769 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:10:31.672809 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:31.672812 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:31.673685 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:31.673686 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:31.674584 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:31.674587 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:31.675109 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:31.675110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:31.675158 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:31.675756 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:10:31.675761 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:10:31.677502 (+ 1741us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:32971: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":35}
W20260926 01:10:31.677688 9547 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:10:31.672463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41841 (local address 127.0.98.62:32971)
0926 01:10:31.672517 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:31.672519 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:31.672526 (+ 7us) server_negotiation.cc:407] Connection header received
0926 01:10:31.672545 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:31.672546 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:31.672557 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:31.672616 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:31.672892 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:31.673603 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:31.674793 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:31.675007 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:31.675072 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:31.677623 (+ 2551us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:41841: BlockingRecv error: recv got EOF from 127.0.98.3:41841 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
W20260926 01:10:31.677702 9543 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 26 01:10:32 dist-test-slave-7b22 krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:10:32 dist-test-slave-7b22 krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:10:32.438866 9548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:32.431419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32971 (local address 127.0.98.1:58833)
0926 01:10:32.431660 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:32.431676 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:32.431766 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:32.432248 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:32.432251 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:32.432508 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:10:32.432758 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:32.432766 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:32.433658 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:32.433660 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:32.434443 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:32.434448 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:32.435273 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:32.435275 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:32.435373 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:32.436410 (+ 1037us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:10:32.436415 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:10:32.438777 (+ 2362us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:32971: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":80,"threads_started":1}
W20260926 01:10:32.439002 9549 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:10:32.431798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58833 (local address 127.0.98.62:32971)
0926 01:10:32.431962 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:32.431964 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:32.431974 (+ 10us) server_negotiation.cc:407] Connection header received
0926 01:10:32.432035 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:32.432036 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:32.432049 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:32.432117 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:32.432944 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:32.433527 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:32.434601 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:32.435105 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:32.435187 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:32.438916 (+ 3729us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:58833: BlockingRecv error: recv got EOF from 127.0.98.1:58833 (error 108)
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":77,"threads_started":1}
Sep 26 01:10:32 dist-test-slave-7b22 krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:10:32 dist-test-slave-7b22 krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:10:32.558554 9550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:32.553444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32971 (local address 127.0.98.2:37113)
0926 01:10:32.553579 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:32.553593 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:32.553670 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:32.553819 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:32.553844 (+ 25us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:32.554018 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:10:32.554065 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:32.554067 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:32.554745 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:32.554746 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:32.555370 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:32.555372 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:32.555710 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:32.555711 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:32.555764 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:32.556231 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:10:32.556236 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:10:32.558449 (+ 2213us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:32971: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":65,"threads_started":1}
W20260926 01:10:32.558599 9549 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:10:32.553534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37113 (local address 127.0.98.62:32971)
0926 01:10:32.553622 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:32.553624 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:32.553648 (+ 24us) server_negotiation.cc:407] Connection header received
0926 01:10:32.553729 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:32.553730 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:32.553747 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:32.553803 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:32.554139 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:32.554661 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:32.555458 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:32.555645 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:32.555703 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:32.558542 (+ 2839us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:37113: BlockingRecv error: recv got EOF from 127.0.98.2:37113 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 26 01:10:32 dist-test-slave-7b22 krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:10:32 dist-test-slave-7b22 krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:10:32.682644 9551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:32.678115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32971 (local address 127.0.98.3:60687)
0926 01:10:32.678249 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:32.678260 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:32.678300 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:32.678445 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:32.678445 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:32.678597 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:10:32.678645 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:32.678646 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:32.679313 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:32.679315 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:32.679879 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:32.679881 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:32.680266 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:32.680268 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:32.680320 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:32.680860 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:10:32.680867 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:10:32.682576 (+ 1709us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:32971: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":59,"threads_started":1}
W20260926 01:10:32.682775 9549 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:10:32.678220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60687 (local address 127.0.98.62:32971)
0926 01:10:32.678277 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:32.678279 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:32.678319 (+ 40us) server_negotiation.cc:407] Connection header received
0926 01:10:32.678343 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:32.678344 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:32.678352 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:32.678398 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:32.678706 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:32.679212 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:32.679980 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:32.680172 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:32.680227 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:32.682696 (+ 2469us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:60687: BlockingRecv error: recv got EOF from 127.0.98.3:60687 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 26 01:10:33 dist-test-slave-7b22 krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:10:33 dist-test-slave-7b22 krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:10:33.444164 9552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:33.439552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32971 (local address 127.0.98.1:34675)
0926 01:10:33.439673 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:33.439684 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:33.439733 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:33.439875 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:33.439875 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:33.440049 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:10:33.440100 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:33.440102 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:33.440784 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:33.440785 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:33.441333 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:33.441335 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:33.441717 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:33.441718 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:33.441765 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:33.442382 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:10:33.442386 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:10:33.444105 (+ 1719us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:32971: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":54,"threads_started":1}
W20260926 01:10:33.444322 9553 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:10:33.439645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34675 (local address 127.0.98.62:32971)
0926 01:10:33.439754 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:33.439756 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:33.439763 (+ 7us) server_negotiation.cc:407] Connection header received
0926 01:10:33.439784 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:33.439785 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:33.439795 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:33.439854 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:33.440159 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:33.440703 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:33.441433 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:33.441639 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:33.441704 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:33.444250 (+ 2546us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:34675: BlockingRecv error: recv got EOF from 127.0.98.1:34675 (error 108)
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":52,"threads_started":1}
Sep 26 01:10:33 dist-test-slave-7b22 krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:10:33 dist-test-slave-7b22 krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:10:33.563855 9554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:33.559174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32971 (local address 127.0.98.2:44837)
0926 01:10:33.559299 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:33.559310 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:33.559380 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:33.559549 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:33.559550 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:33.559718 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:10:33.559765 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:33.559766 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:33.560451 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:33.560453 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:33.560992 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:33.560994 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:33.561335 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:33.561336 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:33.561388 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:33.561886 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:10:33.561891 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:10:33.563774 (+ 1883us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:32971: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":72,"threads_started":1}
W20260926 01:10:33.563968 9553 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:10:33.559317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44837 (local address 127.0.98.62:32971)
0926 01:10:33.559421 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:33.559423 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:33.559431 (+ 8us) server_negotiation.cc:407] Connection header received
0926 01:10:33.559467 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:33.559468 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:33.559479 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:33.559526 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:33.559848 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:33.560363 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:33.561083 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:33.561269 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:33.561326 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:33.563912 (+ 2586us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:44837: BlockingRecv error: recv got EOF from 127.0.98.2:44837 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 26 01:10:33 dist-test-slave-7b22 krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:10:33 dist-test-slave-7b22 krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:10:33.687808 9555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:33.683227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32971 (local address 127.0.98.3:54689)
0926 01:10:33.683417 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:33.683430 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:33.683495 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:33.683657 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:33.683658 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:33.683810 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:10:33.683857 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:33.683859 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:33.684537 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:33.684539 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:33.685074 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:33.685075 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:33.685433 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:33.685434 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:33.685470 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:33.686053 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:10:33.686058 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:10:33.687749 (+ 1691us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:32971: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":89,"threads_started":1}
W20260926 01:10:33.688000 9553 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:10:33.683451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54689 (local address 127.0.98.62:32971)
0926 01:10:33.683540 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:33.683542 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:33.683551 (+ 9us) server_negotiation.cc:407] Connection header received
0926 01:10:33.683575 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:33.683576 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:33.683587 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:33.683643 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:33.683935 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:33.684442 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:33.685178 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:33.685370 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:33.685432 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:33.687933 (+ 2501us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:54689: BlockingRecv error: recv got EOF from 127.0.98.3:54689 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 26 01:10:34 dist-test-slave-7b22 krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:10:34 dist-test-slave-7b22 krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:10:34.449270 9556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:34.444636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32971 (local address 127.0.98.1:46409)
0926 01:10:34.444761 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:34.444772 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:34.444865 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:34.445007 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:34.445008 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:34.445158 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:10:34.445204 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:34.445205 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:34.445907 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:34.445908 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:34.446474 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:34.446476 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:34.446829 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:34.446831 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:34.446891 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:34.447337 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:10:34.447343 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:10:34.449208 (+ 1865us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:32971: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":65,"threads_started":1}
W20260926 01:10:34.449365 9557 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:10:34.444733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46409 (local address 127.0.98.62:32971)
0926 01:10:34.444865 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:34.444867 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:34.444903 (+ 36us) server_negotiation.cc:407] Connection header received
0926 01:10:34.444929 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:34.444930 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:34.444942 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:34.444991 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:34.445259 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:34.445751 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:34.446560 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:34.446752 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:34.446810 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:34.449294 (+ 2484us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:46409: BlockingRecv error: recv got EOF from 127.0.98.1:46409 (error 108)
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":63,"threads_started":1}
Sep 26 01:10:34 dist-test-slave-7b22 krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:10:34 dist-test-slave-7b22 krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:10:34.569271 9558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:34.564450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32971 (local address 127.0.98.2:47889)
0926 01:10:34.564580 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:34.564593 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:34.564664 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:34.564826 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:34.564827 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:34.565003 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:10:34.565050 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:34.565052 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:34.565709 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:34.565711 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:34.566256 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:34.566258 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:34.566595 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:34.566596 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:34.566646 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:34.567171 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:10:34.567176 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:10:34.569199 (+ 2023us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:32971: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":66,"threads_started":1}
W20260926 01:10:34.569446 9557 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:10:34.564568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47889 (local address 127.0.98.62:32971)
0926 01:10:34.564633 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:34.564635 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:34.564646 (+ 11us) server_negotiation.cc:407] Connection header received
0926 01:10:34.564733 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:34.564744 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:34.564757 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:34.564805 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:34.565138 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:34.565632 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:34.566338 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:34.566531 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:34.566588 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:34.569359 (+ 2771us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:47889: BlockingRecv error: recv got EOF from 127.0.98.2:47889 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 26 01:10:34 dist-test-slave-7b22 krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:10:34 dist-test-slave-7b22 krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:10:34.692947 9559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:34.688240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32971 (local address 127.0.98.3:48525)
0926 01:10:34.688390 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:34.688399 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:34.688450 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:34.688581 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:34.688582 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:34.688789 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:10:34.688841 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:34.688843 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:34.689542 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:34.689543 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:34.690220 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:34.690222 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:34.690601 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:34.690602 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:34.690651 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:34.691099 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:10:34.691103 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:10:34.692873 (+ 1770us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:32971: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":71,"threads_started":1}
W20260926 01:10:34.693038 9557 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:10:34.688334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48525 (local address 127.0.98.62:32971)
0926 01:10:34.688381 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:34.688383 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:34.688437 (+ 54us) server_negotiation.cc:407] Connection header received
0926 01:10:34.688490 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:34.688491 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:34.688503 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:34.688571 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:34.688921 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:34.689439 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:34.690323 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:34.690511 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:34.690571 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:34.692977 (+ 2406us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:48525: BlockingRecv error: recv got EOF from 127.0.98.3:48525 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Sep 26 01:10:35 dist-test-slave-7b22 krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:10:35 dist-test-slave-7b22 krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:10:35.454411 9560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:35.449792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32971 (local address 127.0.98.1:52675)
0926 01:10:35.450005 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:35.450019 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:35.450097 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:35.450287 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:35.450288 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:35.450442 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:10:35.450492 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:35.450494 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:35.451120 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:35.451121 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:35.451654 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:35.451655 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:35.452063 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:35.452064 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:35.452105 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:35.452566 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:10:35.452571 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:10:35.454333 (+ 1762us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:32971: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":103,"threads_started":1}
W20260926 01:10:35.454519 9561 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:10:35.449978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52675 (local address 127.0.98.62:32971)
0926 01:10:35.450129 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:35.450131 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:35.450141 (+ 10us) server_negotiation.cc:407] Connection header received
0926 01:10:35.450169 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:35.450170 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:35.450180 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:35.450241 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:35.450562 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:35.451052 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:35.451731 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:35.451984 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:35.452048 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:35.454436 (+ 2388us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:52675: BlockingRecv error: recv got EOF from 127.0.98.1:52675 (error 108)
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":73,"threads_started":1}
Sep 26 01:10:35 dist-test-slave-7b22 krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:10:35 dist-test-slave-7b22 krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:10:35.576189 9562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:35.569745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32971 (local address 127.0.98.2:34853)
0926 01:10:35.569884 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:35.569896 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:35.569966 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:35.571220 (+ 1254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:35.571221 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:35.571414 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:10:35.571504 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:35.571508 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:35.572477 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:35.572479 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:35.573265 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:35.573270 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:35.573750 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:35.573752 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:35.573803 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:35.574261 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:10:35.574265 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:10:35.576126 (+ 1861us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:32971: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":82,"threads_started":1}
W20260926 01:10:35.576318 9561 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:10:35.571032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34853 (local address 127.0.98.62:32971)
0926 01:10:35.571077 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:35.571079 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:35.571089 (+ 10us) server_negotiation.cc:407] Connection header received
0926 01:10:35.571125 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:35.571126 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:35.571139 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:35.571193 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:35.571589 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:35.572374 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:35.573410 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:35.573688 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:35.573741 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:35.576253 (+ 2512us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:34853: BlockingRecv error: recv got EOF from 127.0.98.2:34853 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Sep 26 01:10:35 dist-test-slave-7b22 krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:10:35 dist-test-slave-7b22 krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:10:35.698227 9563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:35.693505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32971 (local address 127.0.98.3:55673)
0926 01:10:35.693646 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:35.693657 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:35.693723 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:35.693918 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:35.693920 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:35.694121 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:10:35.694178 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:35.694181 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:35.694858 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:35.694859 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:35.695390 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:35.695392 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:35.695887 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:35.695888 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:35.695935 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:35.696377 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:10:35.696381 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:10:35.698163 (+ 1782us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:32971: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":69,"threads_started":1}
W20260926 01:10:35.698370 9561 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:10:35.693637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55673 (local address 127.0.98.62:32971)
0926 01:10:35.693705 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:35.693708 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:35.693718 (+ 10us) server_negotiation.cc:407] Connection header received
0926 01:10:35.693781 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:35.693782 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:35.693797 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:35.693897 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:35.694254 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:35.694790 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:35.695528 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:35.695763 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:35.695875 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:35.698303 (+ 2428us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:55673: BlockingRecv error: recv got EOF from 127.0.98.3:55673 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 26 01:10:36 dist-test-slave-7b22 krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:10:36 dist-test-slave-7b22 krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:10:36.464103 9564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:36.454932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32971 (local address 127.0.98.1:42113)
0926 01:10:36.455065 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:36.455075 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:36.455128 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:36.455256 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:36.455257 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:36.455497 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:10:36.455544 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:36.455547 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:36.456299 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:36.456300 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:36.456842 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:36.456844 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:36.457212 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:36.457213 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:36.457263 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:36.462177 (+ 4914us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:10:36.462184 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:10:36.464027 (+ 1843us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:32971: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":79,"threads_started":1}
W20260926 01:10:36.464260 9565 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:10:36.455031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42113 (local address 127.0.98.62:32971)
0926 01:10:36.455140 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:36.455142 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:36.455165 (+ 23us) server_negotiation.cc:407] Connection header received
0926 01:10:36.455185 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:36.455186 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:36.455195 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:36.455242 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:36.455629 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:36.456148 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:36.456928 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:36.457132 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:36.457185 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:36.464135 (+ 6950us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:42113: BlockingRecv error: recv got EOF from 127.0.98.1:42113 (error 108)
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":47,"threads_started":1}
Sep 26 01:10:36 dist-test-slave-7b22 krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:10:36 dist-test-slave-7b22 krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:10:36.581185 9566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:36.576712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32971 (local address 127.0.98.2:60639)
0926 01:10:36.576830 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:36.576840 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:36.576892 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:36.577081 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:36.577082 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:36.577258 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:10:36.577304 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:36.577305 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:36.578010 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:36.578011 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:36.578560 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:36.578562 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:36.578905 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:36.578906 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:36.578959 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:36.579390 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:10:36.579395 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:10:36.581112 (+ 1717us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:32971: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":54,"threads_started":1}
W20260926 01:10:36.581276 9565 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:10:36.576871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60639 (local address 127.0.98.62:32971)
0926 01:10:36.576942 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:36.576944 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:36.576954 (+ 10us) server_negotiation.cc:407] Connection header received
0926 01:10:36.577011 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:36.577012 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:36.577024 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:36.577067 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:36.577389 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:36.577938 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:36.578647 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:36.578837 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:36.578893 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:36.581211 (+ 2318us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:60639: BlockingRecv error: recv got EOF from 127.0.98.2:60639 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 26 01:10:36 dist-test-slave-7b22 krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:10:36 dist-test-slave-7b22 krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:10:36.703152 9567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:36.698782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32971 (local address 127.0.98.3:46357)
0926 01:10:36.698894 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:36.698905 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:36.698971 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:36.699090 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:36.699090 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:36.699249 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:10:36.699295 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:36.699296 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:36.699975 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:36.699975 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:36.700518 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:36.700519 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:36.700849 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:36.700849 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:36.700886 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:36.701295 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:10:36.701299 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:10:36.703093 (+ 1794us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:32971: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":56,"threads_started":1}
W20260926 01:10:36.703241 9565 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:10:36.698859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46357 (local address 127.0.98.62:32971)
0926 01:10:36.698972 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:36.698973 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:36.699002 (+ 29us) server_negotiation.cc:407] Connection header received
0926 01:10:36.699027 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:36.699028 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:36.699040 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:36.699076 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:36.699371 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:36.699860 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:36.700594 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:36.700788 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:36.700842 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:36.703180 (+ 2338us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:46357: BlockingRecv error: recv got EOF from 127.0.98.3:46357 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Sep 26 01:10:37 dist-test-slave-7b22 krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:10:37 dist-test-slave-7b22 krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:10:37.469216 9568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:37.464638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32971 (local address 127.0.98.1:34025)
0926 01:10:37.464786 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:37.464797 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:37.464849 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:37.465034 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:37.465034 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:37.465186 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:10:37.465233 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:37.465234 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:37.465942 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:37.465943 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:37.466647 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:37.466650 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:37.466989 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:37.466990 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:37.467026 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:37.467450 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:10:37.467455 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:10:37.469144 (+ 1689us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:32971: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":70,"threads_started":1}
W20260926 01:10:37.469314 9569 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:10:37.464735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34025 (local address 127.0.98.62:32971)
0926 01:10:37.464893 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:37.464896 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:37.464907 (+ 11us) server_negotiation.cc:407] Connection header received
0926 01:10:37.464936 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:37.464937 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:37.464948 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:37.465021 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:37.465291 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:37.465859 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:37.466723 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:37.466931 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:37.466984 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:37.469244 (+ 2260us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:34025: BlockingRecv error: recv got EOF from 127.0.98.1:34025 (error 108)
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":62,"threads_started":1}
Sep 26 01:10:37 dist-test-slave-7b22 krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:10:37 dist-test-slave-7b22 krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:10:37.586198 9570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:37.581731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32971 (local address 127.0.98.2:35305)
0926 01:10:37.581879 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:37.581891 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:37.581959 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:37.582090 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:37.582091 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:37.582249 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:10:37.582297 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:37.582299 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:37.582915 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:37.582915 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:37.583466 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:37.583467 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:37.583775 (+ 308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:37.583776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:37.583813 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:37.584266 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:10:37.584270 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:10:37.586117 (+ 1847us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:32971: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":57,"threads_started":1}
W20260926 01:10:37.586275 9569 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:10:37.581852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35305 (local address 127.0.98.62:32971)
0926 01:10:37.581901 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:37.581903 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:37.581995 (+ 92us) server_negotiation.cc:407] Connection header received
0926 01:10:37.582020 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:37.582021 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:37.582033 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:37.582072 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:37.582355 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:37.582856 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:37.583537 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:37.583722 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:37.583769 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:37.586218 (+ 2449us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:35305: BlockingRecv error: recv got EOF from 127.0.98.2:35305 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Sep 26 01:10:37 dist-test-slave-7b22 krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:10:37 dist-test-slave-7b22 krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:10:37.708312 9571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:37.703721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32971 (local address 127.0.98.3:53611)
0926 01:10:37.703857 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:37.703868 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:37.703922 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:37.704115 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:37.704115 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:37.704267 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:10:37.704313 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:37.704314 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:37.704971 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:37.704972 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:37.705488 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:37.705490 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:37.705940 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:37.705941 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:37.705995 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:37.706444 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:10:37.706448 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:10:37.708218 (+ 1770us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:32971: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":68,"threads_started":1}
W20260926 01:10:37.708389 9569 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:10:37.703852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53611 (local address 127.0.98.62:32971)
0926 01:10:37.703974 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:37.703976 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:37.703987 (+ 11us) server_negotiation.cc:407] Connection header received
0926 01:10:37.704024 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:37.704025 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:37.704037 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:37.704101 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:37.704393 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:37.704897 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:37.705596 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:37.705789 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:37.705861 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:37.708332 (+ 2471us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:53611: BlockingRecv error: recv got EOF from 127.0.98.3:53611 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 26 01:10:38 dist-test-slave-7b22 krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:10:38 dist-test-slave-7b22 krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:10:38.474591 9572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:38.469749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32971 (local address 127.0.98.1:38135)
0926 01:10:38.469907 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:38.469918 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:38.469976 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:38.470214 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:38.470215 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:38.470375 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:10:38.470422 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:38.470424 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:38.471076 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:38.471076 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:38.471624 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:38.471625 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:38.472125 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:38.472126 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:38.472181 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:38.472639 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:10:38.472644 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:10:38.474510 (+ 1866us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:32971: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":110,"threads_started":1}
W20260926 01:10:38.474689 9573 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:10:38.469955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38135 (local address 127.0.98.62:32971)
0926 01:10:38.470091 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:38.470094 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:38.470104 (+ 10us) server_negotiation.cc:407] Connection header received
0926 01:10:38.470128 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:38.470129 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:38.470141 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:38.470200 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:38.470492 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:38.471010 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:38.471742 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:38.472014 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:38.472089 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:38.474625 (+ 2536us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:38135: BlockingRecv error: recv got EOF from 127.0.98.1:38135 (error 108)
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":65,"threads_started":1}
Sep 26 01:10:38 dist-test-slave-7b22 krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:10:38 dist-test-slave-7b22 krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:10:38.591257 9574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:38.586676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32971 (local address 127.0.98.2:52555)
0926 01:10:38.586799 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:38.586810 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:38.586852 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:38.586972 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:38.586973 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:38.587127 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:10:38.587174 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:38.587175 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:38.587852 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:38.587853 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:38.588409 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:38.588411 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:38.588737 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:38.588738 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:38.588782 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:38.589244 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:10:38.589249 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:10:38.591177 (+ 1928us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:32971: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":55,"threads_started":1}
W20260926 01:10:38.591338 9573 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:10:38.586766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52555 (local address 127.0.98.62:32971)
0926 01:10:38.586828 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:38.586830 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:38.586858 (+ 28us) server_negotiation.cc:407] Connection header received
0926 01:10:38.586906 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:38.586907 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:38.586918 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:38.586959 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:38.587245 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:38.587774 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:38.588477 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:38.588677 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:38.588727 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:38.591283 (+ 2556us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:52555: BlockingRecv error: recv got EOF from 127.0.98.2:52555 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Sep 26 01:10:38 dist-test-slave-7b22 krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:10:38 dist-test-slave-7b22 krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:10:38.713371 9575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:38.708920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32971 (local address 127.0.98.3:55585)
0926 01:10:38.709082 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:38.709101 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:38.709175 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:38.709298 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:38.709298 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:38.709481 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:10:38.709527 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:38.709529 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:38.710269 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:38.710270 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:38.710830 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:38.710832 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:38.711150 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:38.711151 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:38.711204 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:38.711646 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:10:38.711650 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:10:38.713282 (+ 1632us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:32971: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":73,"threads_started":1}
W20260926 01:10:38.713438 9573 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:10:38.708962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55585 (local address 127.0.98.62:32971)
0926 01:10:38.709064 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:38.709066 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:38.709175 (+ 109us) server_negotiation.cc:407] Connection header received
0926 01:10:38.709230 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:38.709231 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:38.709244 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:38.709293 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:38.709597 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:38.710193 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:38.710913 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:38.711091 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:38.711145 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:38.713382 (+ 2237us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:55585: BlockingRecv error: recv got EOF from 127.0.98.3:55585 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 26 01:10:39 dist-test-slave-7b22 krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:10:39 dist-test-slave-7b22 krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:10:39.479970 9576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:39.475149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32971 (local address 127.0.98.1:41053)
0926 01:10:39.475302 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:39.475312 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:39.475358 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:39.475641 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:39.475641 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:39.475792 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:10:39.475840 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:39.475841 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:39.476566 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:39.476567 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:39.477106 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:39.477108 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:39.477541 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:39.477542 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:39.477594 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:39.478114 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:10:39.478119 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:10:39.479910 (+ 1791us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:32971: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":73,"threads_started":1}
W20260926 01:10:39.480098 9577 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:10:39.475314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41053 (local address 127.0.98.62:32971)
0926 01:10:39.475500 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:39.475502 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:39.475512 (+ 10us) server_negotiation.cc:407] Connection header received
0926 01:10:39.475538 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:39.475538 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:39.475548 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:39.475629 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:39.475933 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:39.476476 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:39.477205 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:39.477434 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:39.477508 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:39.480015 (+ 2507us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:41053: BlockingRecv error: recv got EOF from 127.0.98.1:41053 (error 108)
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":89,"threads_started":1}
Sep 26 01:10:39 dist-test-slave-7b22 krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:10:39 dist-test-slave-7b22 krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:10:39.596758 9578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:39.591864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32971 (local address 127.0.98.2:51331)
0926 01:10:39.592019 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:39.592029 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:39.592083 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:39.592273 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:39.592274 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:39.592449 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:10:39.592497 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:39.592499 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:39.593349 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:39.593351 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:39.593964 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:39.593967 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:39.594351 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:39.594352 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:39.594404 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:39.594938 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:10:39.594943 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:10:39.596677 (+ 1734us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:32971: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":85,"threads_started":1}
W20260926 01:10:39.596834 9577 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:10:39.591990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51331 (local address 127.0.98.62:32971)
0926 01:10:39.592060 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:39.592062 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:39.592073 (+ 11us) server_negotiation.cc:407] Connection header received
0926 01:10:39.592151 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:39.592152 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:39.592162 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:39.592219 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:39.592596 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:39.593250 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:39.594051 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:39.594259 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:39.594308 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:39.596780 (+ 2472us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:51331: BlockingRecv error: recv got EOF from 127.0.98.2:51331 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 26 01:10:39 dist-test-slave-7b22 krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:10:39 dist-test-slave-7b22 krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:10:39.718453 9579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:39.713962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32971 (local address 127.0.98.3:34689)
0926 01:10:39.714108 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:39.714119 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:39.714180 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:39.714308 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:39.714309 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:39.714479 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:10:39.714533 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:39.714535 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:39.715197 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:39.715198 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:39.715735 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:39.715737 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:39.716049 (+ 312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:39.716050 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:39.716100 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:39.716599 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:10:39.716604 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:10:39.718387 (+ 1783us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:32971: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":63,"threads_started":1}
W20260926 01:10:39.718598 9577 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:10:39.714047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34689 (local address 127.0.98.62:32971)
0926 01:10:39.714110 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:39.714113 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:39.714161 (+ 48us) server_negotiation.cc:407] Connection header received
0926 01:10:39.714232 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:39.714233 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:39.714247 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:39.714294 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:39.714601 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:39.715120 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:39.715806 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:39.715987 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:39.716050 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:39.718524 (+ 2474us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:34689: BlockingRecv error: recv got EOF from 127.0.98.3:34689 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Sep 26 01:10:40 dist-test-slave-7b22 krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:10:40 dist-test-slave-7b22 krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:10:40.485435 9580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:40.480508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32971 (local address 127.0.98.1:42333)
0926 01:10:40.480668 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:40.480679 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:40.480739 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:40.480963 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:40.480965 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:40.481131 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:10:40.481180 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:40.481182 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:40.481964 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:40.481965 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:40.482497 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:40.482499 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:40.482916 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:40.482917 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:40.482972 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:40.483463 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:10:40.483467 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:10:40.485358 (+ 1891us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:32971: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":71,"threads_started":1}
W20260926 01:10:40.485558 9581 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:10:40.480625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42333 (local address 127.0.98.62:32971)
0926 01:10:40.480727 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:40.480729 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:40.480738 (+ 9us) server_negotiation.cc:407] Connection header received
0926 01:10:40.480790 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:40.480791 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:40.480801 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:40.480865 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:40.481285 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:40.481869 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:40.482594 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:40.482813 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:40.482878 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:40.485486 (+ 2608us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:42333: BlockingRecv error: recv got EOF from 127.0.98.1:42333 (error 108)
Metrics: {"server-negotiator.queue_time_us":82,"thread_start_us":43,"threads_started":1}
Sep 26 01:10:40 dist-test-slave-7b22 krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:10:40 dist-test-slave-7b22 krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:10:40.602195 9582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:40.597316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32971 (local address 127.0.98.2:56205)
0926 01:10:40.597457 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:40.597471 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:40.597539 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:40.597648 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:40.597648 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:40.597808 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:10:40.597898 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:40.597900 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:40.598613 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:40.598614 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:40.599202 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:40.599204 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:40.599680 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:40.599681 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:40.599735 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:40.600263 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:10:40.600268 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:10:40.602129 (+ 1861us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:32971: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":75,"threads_started":1}
W20260926 01:10:40.602345 9581 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:10:40.597451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56205 (local address 127.0.98.62:32971)
0926 01:10:40.597496 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:40.597498 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:40.597521 (+ 23us) server_negotiation.cc:407] Connection header received
0926 01:10:40.597585 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:40.597586 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:40.597595 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:40.597634 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:40.598008 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:40.598503 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:40.599361 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:40.599580 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:40.599650 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:40.602281 (+ 2631us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:56205: BlockingRecv error: recv got EOF from 127.0.98.2:56205 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Sep 26 01:10:40 dist-test-slave-7b22 krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:10:40 dist-test-slave-7b22 krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:10:40.723703 9583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:40.718927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32971 (local address 127.0.98.3:53565)
0926 01:10:40.719063 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:40.719074 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:40.719136 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:40.719297 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:40.719298 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:40.719490 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:10:40.719544 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:40.719546 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:40.720221 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:40.720222 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:40.720836 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:40.720838 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:40.721210 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:40.721211 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:40.721275 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:40.721748 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:10:40.721752 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:10:40.723620 (+ 1868us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:32971: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":61,"threads_started":1}
W20260926 01:10:40.723780 9581 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:10:40.719071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53565 (local address 127.0.98.62:32971)
0926 01:10:40.719126 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:40.719128 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:40.719136 (+ 8us) server_negotiation.cc:407] Connection header received
0926 01:10:40.719183 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:40.719184 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:40.719195 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:40.719243 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:40.719620 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:40.720131 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:40.720919 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:40.721117 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:40.721212 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:40.723724 (+ 2512us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:53565: BlockingRecv error: recv got EOF from 127.0.98.3:53565 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Sep 26 01:10:41 dist-test-slave-7b22 krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:10:41 dist-test-slave-7b22 krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:10:41.496651 9584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:41.486106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32971 (local address 127.0.98.1:60721)
0926 01:10:41.486232 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:41.486243 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:41.486296 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:41.486561 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:41.486561 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:41.486739 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:10:41.486785 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:41.486786 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:41.487509 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:41.487510 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:41.488078 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:41.488080 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:41.488529 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:41.488530 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:41.488609 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:41.494665 (+ 6056us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:10:41.494670 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:10:41.496560 (+ 1890us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:32971: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":62,"threads_started":1}
W20260926 01:10:41.496748 9585 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:10:41.486139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60721 (local address 127.0.98.62:32971)
0926 01:10:41.486288 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:41.486289 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:41.486301 (+ 12us) server_negotiation.cc:407] Connection header received
0926 01:10:41.486348 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:41.486349 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:41.486359 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:41.486534 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:41.486873 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:41.487406 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:41.488171 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:41.488434 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:41.488514 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:41.496686 (+ 8172us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:60721: BlockingRecv error: recv got EOF from 127.0.98.1:60721 (error 108)
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":68,"threads_started":1}
Sep 26 01:10:41 dist-test-slave-7b22 krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:10:41 dist-test-slave-7b22 krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:10:41.607532 9586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:41.602750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32971 (local address 127.0.98.2:42785)
0926 01:10:41.602902 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:41.602913 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:41.602983 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:41.603115 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:41.603115 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:41.603274 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:10:41.603321 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:41.603323 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:41.603995 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:41.603997 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:41.604553 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:41.604555 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:41.605065 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:41.605066 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:41.605118 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:41.605608 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:10:41.605613 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:10:41.607470 (+ 1857us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:32971: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":73,"threads_started":1}
W20260926 01:10:41.607666 9585 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:10:41.602853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42785 (local address 127.0.98.62:32971)
0926 01:10:41.602913 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:41.602915 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:41.603000 (+ 85us) server_negotiation.cc:407] Connection header received
0926 01:10:41.603029 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:41.603030 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:41.603040 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:41.603087 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:41.603375 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:41.603867 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:41.604666 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:41.604919 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:41.604991 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:41.607603 (+ 2612us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:42785: BlockingRecv error: recv got EOF from 127.0.98.2:42785 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Sep 26 01:10:41 dist-test-slave-7b22 krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:10:41 dist-test-slave-7b22 krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:10:41.729087 9587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:41.724232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32971 (local address 127.0.98.3:58373)
0926 01:10:41.724397 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:41.724409 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:41.724493 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:41.724658 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:41.724660 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:41.724825 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:10:41.724872 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:41.724874 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:41.725598 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:41.725599 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:41.726201 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:41.726204 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:41.726595 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:41.726596 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:41.726644 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:41.727146 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:10:41.727151 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:10:41.729011 (+ 1860us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:32971: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":74,"threads_started":1}
W20260926 01:10:41.729243 9585 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:10:41.724393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58373 (local address 127.0.98.62:32971)
0926 01:10:41.724448 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:41.724450 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:41.724476 (+ 26us) server_negotiation.cc:407] Connection header received
0926 01:10:41.724545 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:41.724546 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:41.724557 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:41.724606 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:41.724972 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:41.725510 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:41.726291 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:41.726517 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:41.726576 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:41.729160 (+ 2584us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:58373: BlockingRecv error: recv got EOF from 127.0.98.3:58373 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Sep 26 01:10:42 dist-test-slave-7b22 krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:10:42 dist-test-slave-7b22 krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:10:42.503336 9588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:42.497305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32971 (local address 127.0.98.1:33801)
0926 01:10:42.497481 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:42.497493 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:42.497551 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:42.497740 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:42.497741 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:42.497971 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:10:42.498031 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:42.498033 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:42.498961 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:42.498963 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:42.499765 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:42.499768 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:42.500253 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:42.500254 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:42.500302 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:42.500985 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:10:42.500991 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:10:42.503252 (+ 2261us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:32971: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":71,"threads_started":1}
W20260926 01:10:42.503517 9589 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:10:42.497396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33801 (local address 127.0.98.62:32971)
0926 01:10:42.497525 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:42.497527 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:42.497542 (+ 15us) server_negotiation.cc:407] Connection header received
0926 01:10:42.497642 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:42.497643 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:42.497657 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:42.497729 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:42.498248 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:42.498837 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:42.499853 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:42.500173 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:42.500243 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:42.503448 (+ 3205us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:33801: BlockingRecv error: recv got EOF from 127.0.98.1:33801 (error 108)
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":62,"threads_started":1}
Sep 26 01:10:42 dist-test-slave-7b22 krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:10:42 dist-test-slave-7b22 krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:10:42.612699 9590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:42.608120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32971 (local address 127.0.98.2:47177)
0926 01:10:42.608273 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:42.608282 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:42.608340 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:42.608468 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:42.608468 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:42.608644 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:10:42.608690 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:42.608692 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:42.609401 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:42.609401 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:42.609977 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:42.609980 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:42.610362 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:42.610363 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:42.610421 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:42.610873 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:10:42.610877 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:10:42.612638 (+ 1761us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:32971: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":72,"threads_started":1}
W20260926 01:10:42.612809 9589 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:10:42.608206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47177 (local address 127.0.98.62:32971)
0926 01:10:42.608262 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:42.608263 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:42.608336 (+ 73us) server_negotiation.cc:407] Connection header received
0926 01:10:42.608394 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:42.608395 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:42.608404 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:42.608455 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:42.608790 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:42.609296 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:42.610070 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:42.610264 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:42.610329 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:42.612746 (+ 2417us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:47177: BlockingRecv error: recv got EOF from 127.0.98.2:47177 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Sep 26 01:10:42 dist-test-slave-7b22 krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:10:42 dist-test-slave-7b22 krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:10:42.734598 9591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:42.729661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32971 (local address 127.0.98.3:42123)
0926 01:10:42.729805 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:42.729844 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:42.729928 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:42.730085 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:42.730086 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:42.730274 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:10:42.730319 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:42.730321 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:42.730951 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:42.730953 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:42.731467 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:42.731469 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:42.731804 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:42.731805 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:42.731860 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:42.732362 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:10:42.732367 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:10:42.734503 (+ 2136us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:32971: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":69,"threads_started":1}
W20260926 01:10:42.734656 9589 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:10:42.729773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42123 (local address 127.0.98.62:32971)
0926 01:10:42.729860 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:42.729863 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:42.729907 (+ 44us) server_negotiation.cc:407] Connection header received
0926 01:10:42.730007 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:42.730008 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:42.730021 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:42.730067 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:42.730393 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:42.730877 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:42.731556 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:42.731744 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:42.731799 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:42.734601 (+ 2802us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:42123: BlockingRecv error: recv got EOF from 127.0.98.3:42123 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 26 01:10:43 dist-test-slave-7b22 krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:10:43 dist-test-slave-7b22 krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:10:43.508637 9592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:43.503795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32971 (local address 127.0.98.1:42315)
0926 01:10:43.503924 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:43.503934 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:43.503980 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:43.504178 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:43.504178 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:43.504336 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:10:43.504386 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:43.504387 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:43.505113 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:43.505114 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:43.505666 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:43.505668 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:43.506040 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:43.506041 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:43.506088 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:43.506541 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:10:43.506546 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:10:43.508561 (+ 2015us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:32971: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":62,"threads_started":1}
W20260926 01:10:43.508751 9593 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:10:43.503907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42315 (local address 127.0.98.62:32971)
0926 01:10:43.504035 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:43.504037 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:43.504044 (+ 7us) server_negotiation.cc:407] Connection header received
0926 01:10:43.504064 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:43.504065 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:43.504074 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:43.504136 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:43.504477 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:43.505012 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:43.505752 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:43.505977 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:43.506033 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:43.508679 (+ 2646us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:42315: BlockingRecv error: recv got EOF from 127.0.98.1:42315 (error 108)
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":72,"threads_started":1}
Sep 26 01:10:43 dist-test-slave-7b22 krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:10:43 dist-test-slave-7b22 krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:10:43.618044 9594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:43.613250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32971 (local address 127.0.98.2:50727)
0926 01:10:43.613369 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:43.613382 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:43.613451 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:43.613584 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:43.613585 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:43.613796 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:10:43.613877 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:43.613879 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:43.614560 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:43.614561 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:43.615128 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:43.615131 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:43.615494 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:43.615495 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:43.615546 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:43.615985 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:10:43.615990 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:10:43.617925 (+ 1935us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:32971: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":65,"threads_started":1}
W20260926 01:10:43.618106 9593 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:10:43.613337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50727 (local address 127.0.98.62:32971)
0926 01:10:43.613384 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:43.613386 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:43.613451 (+ 65us) server_negotiation.cc:407] Connection header received
0926 01:10:43.613508 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:43.613509 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:43.613523 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:43.613582 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:43.613942 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:43.614467 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:43.615215 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:43.615406 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:43.615461 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:43.618055 (+ 2594us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:50727: BlockingRecv error: recv got EOF from 127.0.98.2:50727 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Sep 26 01:10:43 dist-test-slave-7b22 krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:10:43 dist-test-slave-7b22 krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:10:43.739509 9595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:43.735141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32971 (local address 127.0.98.3:53903)
0926 01:10:43.735361 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:43.735374 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:43.735448 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:43.735589 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:43.735591 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:43.735755 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:10:43.735800 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:43.735801 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:43.736451 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:43.736452 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:43.736988 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:43.736990 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:43.737307 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:43.737308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:43.737359 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:43.737793 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:10:43.737798 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:10:43.739431 (+ 1633us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:32971: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":123,"threads_started":1}
W20260926 01:10:43.739583 9593 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:10:43.735285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53903 (local address 127.0.98.62:32971)
0926 01:10:43.735368 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:43.735370 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:43.735440 (+ 70us) server_negotiation.cc:407] Connection header received
0926 01:10:43.735508 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:43.735509 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:43.735522 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:43.735574 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:43.735877 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:43.736372 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:43.737065 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:43.737245 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:43.737299 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:43.739529 (+ 2230us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:53903: BlockingRecv error: recv got EOF from 127.0.98.3:53903 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 26 01:10:44 dist-test-slave-7b22 krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:10:44 dist-test-slave-7b22 krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:10:44.513710 9596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:44.509264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32971 (local address 127.0.98.1:47337)
0926 01:10:44.509407 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:44.509419 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:44.509493 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:44.509636 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:44.509637 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:44.509791 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:10:44.509855 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:44.509857 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:44.510493 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:44.510494 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:44.511050 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:44.511052 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:44.511346 (+ 294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:44.511346 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:44.511383 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:44.511861 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:10:44.511866 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:10:44.513611 (+ 1745us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:32971: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":86,"threads_started":1}
W20260926 01:10:44.513780 9597 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:10:44.509375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47337 (local address 127.0.98.62:32971)
0926 01:10:44.509481 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:44.509483 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:44.509494 (+ 11us) server_negotiation.cc:407] Connection header received
0926 01:10:44.509551 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:44.509552 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:44.509561 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:44.509623 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:44.509919 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:44.510421 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:44.511113 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:44.511297 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:44.511342 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:44.513718 (+ 2376us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:47337: BlockingRecv error: recv got EOF from 127.0.98.1:47337 (error 108)
Metrics: {"server-negotiator.queue_time_us":85,"thread_start_us":57,"threads_started":1}
Sep 26 01:10:44 dist-test-slave-7b22 krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:10:44 dist-test-slave-7b22 krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:10:44.623318 9598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:44.618574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32971 (local address 127.0.98.2:53119)
0926 01:10:44.618746 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:44.618758 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:44.618829 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:44.618977 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:44.618978 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:44.619147 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:10:44.619193 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:44.619195 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:44.619860 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:44.619861 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:44.620426 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:44.620428 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:44.620847 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:44.620848 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:44.620921 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:44.621497 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:10:44.621502 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:10:44.623253 (+ 1751us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:32971: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":74,"threads_started":1}
W20260926 01:10:44.623464 9597 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:10:44.618754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53119 (local address 127.0.98.62:32971)
0926 01:10:44.618813 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:44.618816 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:44.618824 (+ 8us) server_negotiation.cc:407] Connection header received
0926 01:10:44.618903 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:44.618904 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:44.618915 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:44.618961 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:44.619270 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:44.619782 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:44.620518 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:44.620732 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:44.620800 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:44.623388 (+ 2588us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:53119: BlockingRecv error: recv got EOF from 127.0.98.2:53119 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 26 01:10:44 dist-test-slave-7b22 krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:10:44 dist-test-slave-7b22 krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:10:44.744776 9599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:44.740136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32971 (local address 127.0.98.3:35947)
0926 01:10:44.740303 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:44.740314 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:44.740383 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:44.740514 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:44.740515 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:44.740675 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:10:44.740723 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:44.740725 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:44.741433 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:44.741434 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:44.742032 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:44.742036 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:44.742405 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:44.742405 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:44.742460 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:44.742885 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:10:44.742890 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:10:44.744684 (+ 1794us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:32971: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":75,"threads_started":1}
W20260926 01:10:44.744856 9597 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:10:44.740239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35947 (local address 127.0.98.62:32971)
0926 01:10:44.740288 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:44.740290 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:44.740379 (+ 89us) server_negotiation.cc:407] Connection header received
0926 01:10:44.740441 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:44.740442 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:44.740454 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:44.740497 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:44.740795 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:44.741346 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:44.742146 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:44.742347 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:44.742405 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:44.744799 (+ 2394us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:35947: BlockingRecv error: recv got EOF from 127.0.98.3:35947 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Sep 26 01:10:45 dist-test-slave-7b22 krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:10:45 dist-test-slave-7b22 krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:10:45.519038 9600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:45.514300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32971 (local address 127.0.98.1:34299)
0926 01:10:45.514430 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:45.514441 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:45.514509 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:45.514645 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:45.514646 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:45.514807 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:10:45.514857 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:45.514858 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:45.515514 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:45.515515 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:45.516088 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:45.516090 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:45.516469 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:45.516470 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:45.516518 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:45.516982 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:10:45.516988 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:10:45.518955 (+ 1967us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:32971: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":69,"threads_started":1}
W20260926 01:10:45.519138 9601 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:10:45.514395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34299 (local address 127.0.98.62:32971)
0926 01:10:45.514509 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:45.514511 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:45.514538 (+ 27us) server_negotiation.cc:407] Connection header received
0926 01:10:45.514571 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:45.514572 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:45.514588 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:45.514639 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:45.514916 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:45.515416 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:45.516178 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:45.516383 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:45.516438 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:45.519067 (+ 2629us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:34299: BlockingRecv error: recv got EOF from 127.0.98.1:34299 (error 108)
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":51,"threads_started":1}
Sep 26 01:10:45 dist-test-slave-7b22 krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:10:45 dist-test-slave-7b22 krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:10:45.628453 9602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:45.623831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32971 (local address 127.0.98.2:48927)
0926 01:10:45.623984 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:45.623995 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:45.624063 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:45.624207 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:45.624208 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:45.624365 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:10:45.624412 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:45.624414 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:45.625124 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:45.625125 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:45.625680 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:45.625682 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:45.626091 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:45.626092 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:45.626151 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:45.626640 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:10:45.626644 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:10:45.628389 (+ 1745us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:32971: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":63,"threads_started":1}
W20260926 01:10:45.628662 9601 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:10:45.623906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48927 (local address 127.0.98.62:32971)
0926 01:10:45.623999 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:45.624001 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:45.624056 (+ 55us) server_negotiation.cc:407] Connection header received
0926 01:10:45.624112 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:45.624113 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:45.624124 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:45.624193 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:45.624505 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:45.625018 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:45.625760 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:45.625997 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:45.626053 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:45.628560 (+ 2507us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:48927: BlockingRecv error: recv got EOF from 127.0.98.2:48927 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Sep 26 01:10:45 dist-test-slave-7b22 krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:10:45 dist-test-slave-7b22 krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:10:45.750195 9603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:45.745301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32971 (local address 127.0.98.3:50945)
0926 01:10:45.745475 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:45.745487 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:45.745570 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:45.745735 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:45.745736 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:45.745926 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:10:45.745975 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:45.745977 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:45.746687 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:45.746688 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:45.747251 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:45.747254 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:45.747611 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:45.747612 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:45.747658 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:45.748245 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:10:45.748249 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:10:45.750132 (+ 1883us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:32971: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":82,"threads_started":1}
W20260926 01:10:45.750310 9601 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:10:45.745408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50945 (local address 127.0.98.62:32971)
0926 01:10:45.745482 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:45.745484 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:45.745560 (+ 76us) server_negotiation.cc:407] Connection header received
0926 01:10:45.745627 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:45.745628 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:45.745640 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:45.745689 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:45.746082 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:45.746592 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:45.747348 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:45.747541 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:45.747595 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:45.750246 (+ 2651us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:50945: BlockingRecv error: recv got EOF from 127.0.98.3:50945 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 26 01:10:46 dist-test-slave-7b22 krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:10:46 dist-test-slave-7b22 krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:10:46.527709 9604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:46.519609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32971 (local address 127.0.98.1:60027)
0926 01:10:46.519736 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:46.519748 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:46.519803 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:46.520044 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:46.520045 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:46.520213 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:10:46.520261 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:46.520263 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:46.520912 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:46.520913 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:46.521534 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:46.521537 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:46.521915 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:46.521916 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:46.521956 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:46.525907 (+ 3951us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:10:46.525914 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:10:46.527648 (+ 1734us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:32971: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":62,"threads_started":1}
W20260926 01:10:46.527863 9605 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:10:46.519711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60027 (local address 127.0.98.62:32971)
0926 01:10:46.519837 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:46.519839 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:46.519847 (+ 8us) server_negotiation.cc:407] Connection header received
0926 01:10:46.519871 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:46.519871 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:46.519881 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:46.519952 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:46.520335 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:46.520839 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:46.521619 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:46.521853 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:46.521911 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:46.527789 (+ 5878us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:60027: BlockingRecv error: recv got EOF from 127.0.98.1:60027 (error 108)
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":67,"threads_started":1}
Sep 26 01:10:46 dist-test-slave-7b22 krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:10:46 dist-test-slave-7b22 krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:10:46.633724 9606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:46.629033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32971 (local address 127.0.98.2:60507)
0926 01:10:46.629197 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:46.629212 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:46.629284 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:46.629423 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:46.629424 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:46.629594 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:10:46.629642 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:46.629643 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:46.630344 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:46.630345 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:46.630908 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:46.630910 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:46.631320 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:46.631321 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:46.631373 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:46.631839 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:10:46.631844 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:10:46.633637 (+ 1793us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:32971: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":65,"threads_started":1}
W20260926 01:10:46.633888 9605 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:10:46.629179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60507 (local address 127.0.98.62:32971)
0926 01:10:46.629248 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:46.629250 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:46.629264 (+ 14us) server_negotiation.cc:407] Connection header received
0926 01:10:46.629343 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:46.629344 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:46.629355 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:46.629405 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:46.629707 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:46.630248 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:46.631038 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:46.631234 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:46.631278 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:46.633804 (+ 2526us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:60507: BlockingRecv error: recv got EOF from 127.0.98.2:60507 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 01:10:46.666568 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 9129
I20260926 01:10:46.670550 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 9281
I20260926 01:10:46.674449 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 9414
I20260926 01:10:46.678248 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 9059
2026-09-26T01:10:46Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17309 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-26T01:10:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:10:46Z Disabled control of system clock
I20260926 01:10:46.704749 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384993574762-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384993574762-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384993574762-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36903
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:38527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36903
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260926 01:10:46.784948 9610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:10:46.785188 9610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:10:46.785207 9610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:10:46.786427 9610 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:10:46.786470 9610 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:10:46.786481 9610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:10:46.786491 9610 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:10:46.786502 9610 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:10:46.787874 9610 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384993574762-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384993574762-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36903
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36903
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9610
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384993574762-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:10:46.788094 9610 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:10:46.788352 9610 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:10:46.793572 9616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:10:46.793553 9610 server_base.cc:1034] running on GCE node
W20260926 01:10:46.793637 9618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:10:46.793543 9615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:10:46.794174 9610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:10:46.794438 9610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:10:46.795603 9610 hybrid_clock.cc:648] HybridClock initialized: now 1790385046795590 us; error 31 us; skew 500 ppm
I20260926 01:10:46.796990 9610 webserver.cc:460] Webserver started at http://127.0.98.62:41387/ using document root <none> and password file <none>
I20260926 01:10:46.797230 9610 fs_manager.cc:359] Metadata directory not provided
I20260926 01:10:46.797281 9610 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:10:46.797392 9610 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:10:46.798491 9610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384993574762-392-0/minicluster-data/master-0/data/instance:
uuid: "92ad9750760b4b73872e11242ffe8723"
format_stamp: "Formatted at 2026-09-26 01:10:46 on dist-test-slave-7b22"
server_key: "7b43ad68ed931ad35493ea7a815d3053"
server_key_iv: "4ab5c15d49a61c32a6af52d9c1cb8b16"
server_key_version: "encryptionkey@0"
I20260926 01:10:46.798815 9610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384993574762-392-0/minicluster-data/master-0/wal/instance:
uuid: "92ad9750760b4b73872e11242ffe8723"
format_stamp: "Formatted at 2026-09-26 01:10:46 on dist-test-slave-7b22"
server_key: "7b43ad68ed931ad35493ea7a815d3053"
server_key_iv: "4ab5c15d49a61c32a6af52d9c1cb8b16"
server_key_version: "encryptionkey@0"
I20260926 01:10:46.800076 9610 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 01:10:46.800864 9624 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:46.801080 9610 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:10:46.801159 9610 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384993574762-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384993574762-392-0/minicluster-data/master-0/wal
uuid: "92ad9750760b4b73872e11242ffe8723"
format_stamp: "Formatted at 2026-09-26 01:10:46 on dist-test-slave-7b22"
server_key: "7b43ad68ed931ad35493ea7a815d3053"
server_key_iv: "4ab5c15d49a61c32a6af52d9c1cb8b16"
server_key_version: "encryptionkey@0"
I20260926 01:10:46.801218 9610 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384993574762-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384993574762-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384993574762-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:10:46.812996 9610 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:10:46.813308 9610 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:10:46.813414 9610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:10:46.817246 9610 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36903
I20260926 01:10:46.817294 9676 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36903 every 8 connection(s)
I20260926 01:10:46.817617 9610 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
I20260926 01:10:46.818274 9677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:10:46.819159 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 9610
I20260926 01:10:46.819242 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384993574762-392-0/minicluster-data/master-0/wal/instance
I20260926 01:10:46.819442 392 external_mini_cluster.cc:1428] Setting key 51698742c7b930f97eb9c050ab771a79
I20260926 01:10:46.821578 9677 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92ad9750760b4b73872e11242ffe8723: Bootstrap starting.
I20260926 01:10:46.822316 9677 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 92ad9750760b4b73872e11242ffe8723: Neither blocks nor log segments found. Creating new log.
I20260926 01:10:46.822649 9677 log.cc:824] T 00000000000000000000000000000000 P 92ad9750760b4b73872e11242ffe8723: Log is configured to *not* fsync() on all Append() calls
I20260926 01:10:46.823282 9677 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92ad9750760b4b73872e11242ffe8723: No bootstrap required, opened a new log
I20260926 01:10:46.825376 9680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:46.820001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40718 (local address 127.0.98.62:36903)
0926 01:10:46.820207 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:46.820216 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:46.820251 (+ 35us) server_negotiation.cc:407] Connection header received
0926 01:10:46.820750 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:46.820760 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:46.820845 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:46.820993 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:46.821180 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:46.822526 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:46.823281 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:46.823850 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:46.823979 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:46.824652 (+ 673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:10:46.824657 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:10:46.824657 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:10:46.824859 (+ 202us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:10:46.824903 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:10:46.824919 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:10:46.824984 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:46.825064 (+ 80us) server_negotiation.cc:299] Negotiation successful
0926 01:10:46.825153 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":82,"threads_started":1}
I20260926 01:10:46.825608 9677 raft_consensus.cc:348] T 00000000000000000000000000000000 P 92ad9750760b4b73872e11242ffe8723 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92ad9750760b4b73872e11242ffe8723" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36903 } }
I20260926 01:10:46.825752 9677 raft_consensus.cc:374] T 00000000000000000000000000000000 P 92ad9750760b4b73872e11242ffe8723 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:10:46.825809 9677 raft_consensus.cc:729] T 00000000000000000000000000000000 P 92ad9750760b4b73872e11242ffe8723 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92ad9750760b4b73872e11242ffe8723, State: Initialized, Role: FOLLOWER
I20260926 01:10:46.825958 9677 consensus_queue.cc:260] T 00000000000000000000000000000000 P 92ad9750760b4b73872e11242ffe8723 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92ad9750760b4b73872e11242ffe8723" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36903 } }
I20260926 01:10:46.826036 9677 raft_consensus.cc:388] T 00000000000000000000000000000000 P 92ad9750760b4b73872e11242ffe8723 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:10:46.826076 9677 raft_consensus.cc:482] T 00000000000000000000000000000000 P 92ad9750760b4b73872e11242ffe8723 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:10:46.826143 9677 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 92ad9750760b4b73872e11242ffe8723 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:10:46.826684 9677 raft_consensus.cc:504] T 00000000000000000000000000000000 P 92ad9750760b4b73872e11242ffe8723 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92ad9750760b4b73872e11242ffe8723" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36903 } }
I20260926 01:10:46.826776 9677 leader_election.cc:302] T 00000000000000000000000000000000 P 92ad9750760b4b73872e11242ffe8723 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 92ad9750760b4b73872e11242ffe8723; no voters:
I20260926 01:10:46.826962 9677 leader_election.cc:288] T 00000000000000000000000000000000 P 92ad9750760b4b73872e11242ffe8723 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:10:46.827049 9682 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 92ad9750760b4b73872e11242ffe8723 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:10:46.827193 9682 raft_consensus.cc:686] T 00000000000000000000000000000000 P 92ad9750760b4b73872e11242ffe8723 [term 1 LEADER]: Becoming Leader. State: Replica: 92ad9750760b4b73872e11242ffe8723, State: Running, Role: LEADER
I20260926 01:10:46.827224 9677 sys_catalog.cc:564] T 00000000000000000000000000000000 P 92ad9750760b4b73872e11242ffe8723 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:10:46.827279 9682 consensus_queue.cc:237] T 00000000000000000000000000000000 P 92ad9750760b4b73872e11242ffe8723 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92ad9750760b4b73872e11242ffe8723" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36903 } }
I20260926 01:10:46.827643 9683 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92ad9750760b4b73872e11242ffe8723 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "92ad9750760b4b73872e11242ffe8723" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92ad9750760b4b73872e11242ffe8723" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36903 } } }
I20260926 01:10:46.827723 9683 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92ad9750760b4b73872e11242ffe8723 [sys.catalog]: This master's current role is: LEADER
I20260926 01:10:46.827865 9684 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92ad9750760b4b73872e11242ffe8723 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 92ad9750760b4b73872e11242ffe8723. Latest consensus state: current_term: 1 leader_uuid: "92ad9750760b4b73872e11242ffe8723" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92ad9750760b4b73872e11242ffe8723" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36903 } } }
I20260926 01:10:46.827905 9690 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:10:46.828495 9690 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:10:46.827976 9684 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92ad9750760b4b73872e11242ffe8723 [sys.catalog]: This master's current role is: LEADER
I20260926 01:10:46.830209 9690 catalog_manager.cc:1347] Generated new cluster ID: c44f429e2b974fa58d4a55bfdd1e326d
I20260926 01:10:46.830256 9690 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 01:10:46.841297 9690 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:10:46.841856 9690 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:10:46.847707 9690 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 92ad9750760b4b73872e11242ffe8723: Generated new TSK 0
I20260926 01:10:46.847944 9690 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 01:10:46.852178 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384993574762-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384993574762-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384993574762-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384993574762-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36903
--builtin_ntp_servers=127.0.98.20:38527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260926 01:10:46.933646 9701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:10:46.933902 9701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:10:46.933925 9701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:10:46.935375 9701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:10:46.935436 9701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:10:46.935467 9701 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 01:10:46.937121 9701 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384993574762-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384993574762-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384993574762-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36903
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9701
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384993574762-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:10:46.937383 9701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:10:46.937688 9701 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:10:46.943464 9707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:10:46.943567 9706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:10:46.943473 9709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:10:46.943825 9701 server_base.cc:1034] running on GCE node
I20260926 01:10:46.944059 9701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:10:46.944319 9701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:10:46.945487 9701 hybrid_clock.cc:648] HybridClock initialized: now 1790385046945473 us; error 29 us; skew 500 ppm
I20260926 01:10:46.946864 9701 webserver.cc:460] Webserver started at http://127.0.98.1:38001/ using document root <none> and password file <none>
I20260926 01:10:46.947108 9701 fs_manager.cc:359] Metadata directory not provided
I20260926 01:10:46.947152 9701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:10:46.947278 9701 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:10:46.948402 9701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384993574762-392-0/minicluster-data/ts-0/data/instance:
uuid: "fe46a54155f9436e95120ae0bdd6dea8"
format_stamp: "Formatted at 2026-09-26 01:10:46 on dist-test-slave-7b22"
server_key: "186453a4921225a8f699865773c04cda"
server_key_iv: "276cb34d0582038a2a2f81ffdcc1513e"
server_key_version: "encryptionkey@0"
I20260926 01:10:46.948762 9701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384993574762-392-0/minicluster-data/ts-0/wal/instance:
uuid: "fe46a54155f9436e95120ae0bdd6dea8"
format_stamp: "Formatted at 2026-09-26 01:10:46 on dist-test-slave-7b22"
server_key: "186453a4921225a8f699865773c04cda"
server_key_iv: "276cb34d0582038a2a2f81ffdcc1513e"
server_key_version: "encryptionkey@0"
I20260926 01:10:46.950090 9701 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 01:10:46.950811 9715 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:46.950945 9701 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 01:10:46.951035 9701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384993574762-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384993574762-392-0/minicluster-data/ts-0/wal
uuid: "fe46a54155f9436e95120ae0bdd6dea8"
format_stamp: "Formatted at 2026-09-26 01:10:46 on dist-test-slave-7b22"
server_key: "186453a4921225a8f699865773c04cda"
server_key_iv: "276cb34d0582038a2a2f81ffdcc1513e"
server_key_version: "encryptionkey@0"
I20260926 01:10:46.951109 9701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384993574762-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384993574762-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384993574762-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:10:46.960610 9701 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:10:46.960933 9701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:10:46.961061 9701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:10:46.961290 9701 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:10:46.961596 9701 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:10:46.961628 9701 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:46.961658 9701 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:10:46.961683 9701 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:46.967408 9701 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34665
I20260926 01:10:46.967456 9828 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34665 every 8 connection(s)
I20260926 01:10:46.967777 9701 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384993574762-392-0/minicluster-data/ts-0/data/info.pb
I20260926 01:10:46.973142 9680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:46.968518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33251 (local address 127.0.98.62:36903)
0926 01:10:46.968582 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:46.968584 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:46.968853 (+ 269us) server_negotiation.cc:407] Connection header received
0926 01:10:46.969371 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:46.969372 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:46.969384 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:46.969448 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:46.970186 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:46.970716 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:46.971806 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:46.972035 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:46.972102 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:46.972895 (+ 793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:10:46.972896 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:10:46.972896 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:10:46.972902 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:10:46.972912 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:10:46.972915 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:10:46.972945 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:46.973055 (+ 110us) server_negotiation.cc:299] Negotiation successful
0926 01:10:46.973078 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 01:10:46.973353 9831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:46.968503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36903 (local address 127.0.98.1:33251)
0926 01:10:46.968695 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:46.968738 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:46.969245 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:46.969509 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:46.969512 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:46.969520 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:10:46.970066 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:46.970072 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:46.970822 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:46.970823 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:46.971709 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:46.971712 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:46.972107 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:46.972108 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:46.972220 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:46.972774 (+ 554us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 01:10:46.972775 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:10:46.972804 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 01:10:46.972828 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:10:46.972954 (+ 126us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:10:46.972956 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:10:46.972959 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 01:10:46.973053 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 01:10:46.973139 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":76,"threads_started":1}
I20260926 01:10:46.973860 9829 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36903
I20260926 01:10:46.973950 9829 heartbeater.cc:464] Registering TS with master...
I20260926 01:10:46.974134 9829 heartbeater.cc:511] Master 127.0.98.62:36903 requested a full tablet report, sending...
I20260926 01:10:46.974634 9641 ts_manager.cc:194] Registered new tserver with Master: fe46a54155f9436e95120ae0bdd6dea8 (127.0.98.1:34665)
I20260926 01:10:46.975260 9641 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:33251
I20260926 01:10:46.976948 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 9701
I20260926 01:10:46.977021 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384993574762-392-0/minicluster-data/ts-0/wal/instance
I20260926 01:10:46.977166 392 external_mini_cluster.cc:1428] Setting key 324e798eb8380f82dcb3ac7d59ea66f0
I20260926 01:10:46.978055 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384993574762-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384993574762-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384993574762-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384993574762-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36903
--builtin_ntp_servers=127.0.98.20:38527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260926 01:10:47.059093 9832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:10:47.059317 9832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:10:47.059340 9832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:10:47.060541 9832 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:10:47.060585 9832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:10:47.060609 9832 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 01:10:47.061980 9832 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384993574762-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384993574762-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384993574762-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36903
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9832
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384993574762-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:10:47.062341 9832 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:10:47.062670 9832 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:10:47.067617 9837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:10:47.067617 9838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:10:47.067684 9840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:10:47.068444 9832 server_base.cc:1034] running on GCE node
I20260926 01:10:47.068686 9832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:10:47.068909 9832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:10:47.070118 9832 hybrid_clock.cc:648] HybridClock initialized: now 1790385047070103 us; error 31 us; skew 500 ppm
I20260926 01:10:47.071420 9832 webserver.cc:460] Webserver started at http://127.0.98.2:45933/ using document root <none> and password file <none>
I20260926 01:10:47.071635 9832 fs_manager.cc:359] Metadata directory not provided
I20260926 01:10:47.071679 9832 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:10:47.071799 9832 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:10:47.072917 9832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384993574762-392-0/minicluster-data/ts-1/data/instance:
uuid: "8a7e653c358b47589cac7944ce9d291c"
format_stamp: "Formatted at 2026-09-26 01:10:47 on dist-test-slave-7b22"
server_key: "f8a39d52993e8473535a9a0b38c67348"
server_key_iv: "3cfe81ce44322362c4576416062d6ec3"
server_key_version: "encryptionkey@0"
I20260926 01:10:47.073278 9832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384993574762-392-0/minicluster-data/ts-1/wal/instance:
uuid: "8a7e653c358b47589cac7944ce9d291c"
format_stamp: "Formatted at 2026-09-26 01:10:47 on dist-test-slave-7b22"
server_key: "f8a39d52993e8473535a9a0b38c67348"
server_key_iv: "3cfe81ce44322362c4576416062d6ec3"
server_key_version: "encryptionkey@0"
I20260926 01:10:47.074651 9832 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 01:10:47.075510 9846 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:47.075680 9832 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 01:10:47.075767 9832 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384993574762-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384993574762-392-0/minicluster-data/ts-1/wal
uuid: "8a7e653c358b47589cac7944ce9d291c"
format_stamp: "Formatted at 2026-09-26 01:10:47 on dist-test-slave-7b22"
server_key: "f8a39d52993e8473535a9a0b38c67348"
server_key_iv: "3cfe81ce44322362c4576416062d6ec3"
server_key_version: "encryptionkey@0"
I20260926 01:10:47.075830 9832 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384993574762-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384993574762-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384993574762-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:10:47.091789 9832 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:10:47.092110 9832 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:10:47.092254 9832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:10:47.092492 9832 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:10:47.092823 9832 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:10:47.092857 9832 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:47.092888 9832 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:10:47.092911 9832 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:47.098934 9832 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45779
I20260926 01:10:47.098973 9959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45779 every 8 connection(s)
I20260926 01:10:47.099294 9832 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384993574762-392-0/minicluster-data/ts-1/data/info.pb
I20260926 01:10:47.102905 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 9832
I20260926 01:10:47.103008 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384993574762-392-0/minicluster-data/ts-1/wal/instance
I20260926 01:10:47.103207 392 external_mini_cluster.cc:1428] Setting key d289b778b314ae597970b02112ec5962
I20260926 01:10:47.104167 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384993574762-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384993574762-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384993574762-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384993574762-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36903
--builtin_ntp_servers=127.0.98.20:38527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260926 01:10:47.104823 9680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:47.100012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47947 (local address 127.0.98.62:36903)
0926 01:10:47.100071 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:47.100073 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:47.100289 (+ 216us) server_negotiation.cc:407] Connection header received
0926 01:10:47.100839 (+ 550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:47.100841 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:47.100853 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:47.100901 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:47.101572 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:47.102107 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:47.103164 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:47.103442 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:47.103534 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:47.104557 (+ 1023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:10:47.104558 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:10:47.104558 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:10:47.104564 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:10:47.104580 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:10:47.104585 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:10:47.104662 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:47.104718 (+ 56us) server_negotiation.cc:299] Negotiation successful
0926 01:10:47.104744 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 01:10:47.105048 9962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:47.099983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36903 (local address 127.0.98.2:47947)
0926 01:10:47.100175 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:47.100209 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:47.100738 (+ 529us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:47.100984 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:47.100988 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:47.100998 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:10:47.101481 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:47.101486 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:47.102193 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:47.102194 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:47.102997 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:47.102999 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:47.103549 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:47.103551 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:47.103720 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:47.104417 (+ 697us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 01:10:47.104418 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:10:47.104449 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 01:10:47.104475 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:10:47.104648 (+ 173us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:10:47.104651 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:10:47.104654 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 01:10:47.104717 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 01:10:47.104823 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":85,"threads_started":1}
I20260926 01:10:47.105536 9960 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36903
I20260926 01:10:47.105631 9960 heartbeater.cc:464] Registering TS with master...
I20260926 01:10:47.105809 9960 heartbeater.cc:511] Master 127.0.98.62:36903 requested a full tablet report, sending...
I20260926 01:10:47.106213 9641 ts_manager.cc:194] Registered new tserver with Master: 8a7e653c358b47589cac7944ce9d291c (127.0.98.2:45779)
I20260926 01:10:47.106649 9641 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:47947
W20260926 01:10:47.201278 9963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:10:47.201510 9963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:10:47.201539 9963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:10:47.202792 9963 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:10:47.202837 9963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:10:47.202873 9963 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 01:10:47.204312 9963 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384993574762-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384993574762-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384993574762-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36903
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9963
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384993574762-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:10:47.204569 9963 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:10:47.204869 9963 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:10:47.210053 9969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:10:47.210054 9971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:10:47.210048 9968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:10:47.210659 9963 server_base.cc:1034] running on GCE node
I20260926 01:10:47.210883 9963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:10:47.211170 9963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:10:47.212342 9963 hybrid_clock.cc:648] HybridClock initialized: now 1790385047212328 us; error 32 us; skew 500 ppm
I20260926 01:10:47.213716 9963 webserver.cc:460] Webserver started at http://127.0.98.3:36337/ using document root <none> and password file <none>
I20260926 01:10:47.213982 9963 fs_manager.cc:359] Metadata directory not provided
I20260926 01:10:47.214036 9963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:10:47.214165 9963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:10:47.215240 9963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384993574762-392-0/minicluster-data/ts-2/data/instance:
uuid: "0143084d529c4b13ab2ab627419e277c"
format_stamp: "Formatted at 2026-09-26 01:10:47 on dist-test-slave-7b22"
server_key: "496aa7fb7f07827159e531bac0c3071b"
server_key_iv: "8b13c85e2929753f2533dfc87f20fa23"
server_key_version: "encryptionkey@0"
I20260926 01:10:47.215601 9963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384993574762-392-0/minicluster-data/ts-2/wal/instance:
uuid: "0143084d529c4b13ab2ab627419e277c"
format_stamp: "Formatted at 2026-09-26 01:10:47 on dist-test-slave-7b22"
server_key: "496aa7fb7f07827159e531bac0c3071b"
server_key_iv: "8b13c85e2929753f2533dfc87f20fa23"
server_key_version: "encryptionkey@0"
I20260926 01:10:47.216872 9963 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 01:10:47.217690 9977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:47.217906 9963 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:10:47.217998 9963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384993574762-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384993574762-392-0/minicluster-data/ts-2/wal
uuid: "0143084d529c4b13ab2ab627419e277c"
format_stamp: "Formatted at 2026-09-26 01:10:47 on dist-test-slave-7b22"
server_key: "496aa7fb7f07827159e531bac0c3071b"
server_key_iv: "8b13c85e2929753f2533dfc87f20fa23"
server_key_version: "encryptionkey@0"
I20260926 01:10:47.218055 9963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384993574762-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384993574762-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384993574762-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:10:47.229928 9963 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:10:47.230242 9963 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:10:47.230350 9963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:10:47.230544 9963 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:10:47.230854 9963 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:10:47.230886 9963 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:47.230919 9963 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:10:47.230942 9963 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:47.236398 9963 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41931
I20260926 01:10:47.236441 10090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41931 every 8 connection(s)
I20260926 01:10:47.236796 9963 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384993574762-392-0/minicluster-data/ts-2/data/info.pb
I20260926 01:10:47.239938 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 9963
I20260926 01:10:47.240044 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384993574762-392-0/minicluster-data/ts-2/wal/instance
I20260926 01:10:47.240280 392 external_mini_cluster.cc:1428] Setting key 63408dd1552da85b73cf1b90eae92d31
I20260926 01:10:47.242390 9680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:47.237530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46369 (local address 127.0.98.62:36903)
0926 01:10:47.237578 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:47.237580 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:47.237807 (+ 227us) server_negotiation.cc:407] Connection header received
0926 01:10:47.238354 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:47.238355 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:47.238364 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:47.238426 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:47.239118 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:47.239642 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:47.241005 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:47.241231 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:47.241299 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:47.242170 (+ 871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:10:47.242171 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:10:47.242171 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:10:47.242175 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:10:47.242190 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:10:47.242194 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:10:47.242235 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:47.242307 (+ 72us) server_negotiation.cc:299] Negotiation successful
0926 01:10:47.242329 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 01:10:47.242607 10093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:47.237495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36903 (local address 127.0.98.3:46369)
0926 01:10:47.237685 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:47.237728 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:47.238261 (+ 533us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:47.238492 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:47.238495 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:47.238504 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:10:47.239004 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:47.239008 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:47.239754 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:47.239755 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:47.240858 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:47.240862 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:47.241313 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:47.241314 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:47.241463 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:47.242037 (+ 574us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 01:10:47.242038 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:10:47.242069 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 01:10:47.242095 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:10:47.242237 (+ 142us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:10:47.242239 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:10:47.242242 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 01:10:47.242293 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 01:10:47.242375 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":82,"threads_started":1}
I20260926 01:10:47.243065 10091 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36903
I20260926 01:10:47.243162 10091 heartbeater.cc:464] Registering TS with master...
I20260926 01:10:47.243350 10091 heartbeater.cc:511] Master 127.0.98.62:36903 requested a full tablet report, sending...
I20260926 01:10:47.243695 9641 ts_manager.cc:194] Registered new tserver with Master: 0143084d529c4b13ab2ab627419e277c (127.0.98.3:41931)
I20260926 01:10:47.244079 9641 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:46369
I20260926 01:10:47.252310 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 01:10:47.254302 9680 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:10:47.253658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40730 (local address 127.0.98.62:36903)
0926 01:10:47.253746 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:47.253748 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:47.253854 (+ 106us) server_negotiation.cc:407] Connection header received
0926 01:10:47.253953 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:47.253954 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:47.253965 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:47.254076 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:47.254227 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40730: BlockingRecv error: recv got EOF from 127.0.0.1:40730 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 01:10:47.254335 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 9701
I20260926 01:10:47.258039 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 9832
I20260926 01:10:47.261540 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 9963
I20260926 01:10:47.265358 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 9610
2026-09-26T01:10:47Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (585 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-26T01:10:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:10:47Z Disabled control of system clock
I20260926 01:10:47.290648 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384993574762-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384993574762-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384993574762-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37905
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:41357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37905
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 01:10:47.372495 10102 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:10:47.372730 10102 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:10:47.372751 10102 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:10:47.373996 10102 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:10:47.374042 10102 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:10:47.374053 10102 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:10:47.374064 10102 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:10:47.374075 10102 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:10:47.375525 10102 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384993574762-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384993574762-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37905
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37905
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10102
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384993574762-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:10:47.375774 10102 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:10:47.376060 10102 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 01:10:47.380972 10102 server_base.cc:1034] running on GCE node
W20260926 01:10:47.380930 10110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:10:47.380862 10108 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:10:47.380862 10107 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:10:47.381336 10102 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:10:47.381657 10102 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:10:47.382826 10102 hybrid_clock.cc:648] HybridClock initialized: now 1790385047382786 us; error 53 us; skew 500 ppm
I20260926 01:10:47.384264 10102 webserver.cc:460] Webserver started at http://127.0.98.62:41097/ using document root <none> and password file <none>
I20260926 01:10:47.384501 10102 fs_manager.cc:359] Metadata directory not provided
I20260926 01:10:47.384588 10102 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:10:47.384719 10102 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:10:47.385782 10102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384993574762-392-0/minicluster-data/master-0/data/instance:
uuid: "eb083140d82e4d3aa44fecb553f1ff70"
format_stamp: "Formatted at 2026-09-26 01:10:47 on dist-test-slave-7b22"
server_key: "73a99196a2b6a6d1db968f9b338ce276"
server_key_iv: "f2e400e6bec45a21a180d9bd49073fba"
server_key_version: "encryptionkey@0"
I20260926 01:10:47.386175 10102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384993574762-392-0/minicluster-data/master-0/wal/instance:
uuid: "eb083140d82e4d3aa44fecb553f1ff70"
format_stamp: "Formatted at 2026-09-26 01:10:47 on dist-test-slave-7b22"
server_key: "73a99196a2b6a6d1db968f9b338ce276"
server_key_iv: "f2e400e6bec45a21a180d9bd49073fba"
server_key_version: "encryptionkey@0"
I20260926 01:10:47.387470 10102 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:10:47.388223 10116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:47.388384 10102 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 01:10:47.388470 10102 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384993574762-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384993574762-392-0/minicluster-data/master-0/wal
uuid: "eb083140d82e4d3aa44fecb553f1ff70"
format_stamp: "Formatted at 2026-09-26 01:10:47 on dist-test-slave-7b22"
server_key: "73a99196a2b6a6d1db968f9b338ce276"
server_key_iv: "f2e400e6bec45a21a180d9bd49073fba"
server_key_version: "encryptionkey@0"
I20260926 01:10:47.388542 10102 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384993574762-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384993574762-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384993574762-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:10:47.392055 10102 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:10:47.392403 10102 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:10:47.392532 10102 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:10:47.396380 10102 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37905
I20260926 01:10:47.396412 10168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37905 every 8 connection(s)
I20260926 01:10:47.396739 10102 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
I20260926 01:10:47.397382 10169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:10:47.400295 10169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb083140d82e4d3aa44fecb553f1ff70: Bootstrap starting.
I20260926 01:10:47.400866 10169 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eb083140d82e4d3aa44fecb553f1ff70: Neither blocks nor log segments found. Creating new log.
I20260926 01:10:47.401103 10169 log.cc:824] T 00000000000000000000000000000000 P eb083140d82e4d3aa44fecb553f1ff70: Log is configured to *not* fsync() on all Append() calls
I20260926 01:10:47.401873 10169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb083140d82e4d3aa44fecb553f1ff70: No bootstrap required, opened a new log
I20260926 01:10:47.403570 10169 raft_consensus.cc:348] T 00000000000000000000000000000000 P eb083140d82e4d3aa44fecb553f1ff70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb083140d82e4d3aa44fecb553f1ff70" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37905 } }
I20260926 01:10:47.403733 10169 raft_consensus.cc:374] T 00000000000000000000000000000000 P eb083140d82e4d3aa44fecb553f1ff70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:10:47.403755 10169 raft_consensus.cc:729] T 00000000000000000000000000000000 P eb083140d82e4d3aa44fecb553f1ff70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb083140d82e4d3aa44fecb553f1ff70, State: Initialized, Role: FOLLOWER
I20260926 01:10:47.403865 10169 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb083140d82e4d3aa44fecb553f1ff70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb083140d82e4d3aa44fecb553f1ff70" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37905 } }
I20260926 01:10:47.403924 10169 raft_consensus.cc:388] T 00000000000000000000000000000000 P eb083140d82e4d3aa44fecb553f1ff70 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:10:47.403951 10169 raft_consensus.cc:482] T 00000000000000000000000000000000 P eb083140d82e4d3aa44fecb553f1ff70 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:10:47.403991 10169 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eb083140d82e4d3aa44fecb553f1ff70 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:10:47.404583 10169 raft_consensus.cc:504] T 00000000000000000000000000000000 P eb083140d82e4d3aa44fecb553f1ff70 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb083140d82e4d3aa44fecb553f1ff70" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37905 } }
I20260926 01:10:47.404688 10169 leader_election.cc:302] T 00000000000000000000000000000000 P eb083140d82e4d3aa44fecb553f1ff70 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eb083140d82e4d3aa44fecb553f1ff70; no voters:
I20260926 01:10:47.404862 10169 leader_election.cc:288] T 00000000000000000000000000000000 P eb083140d82e4d3aa44fecb553f1ff70 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:10:47.404919 10172 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eb083140d82e4d3aa44fecb553f1ff70 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:10:47.405040 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 10102
I20260926 01:10:47.405076 10172 raft_consensus.cc:686] T 00000000000000000000000000000000 P eb083140d82e4d3aa44fecb553f1ff70 [term 1 LEADER]: Becoming Leader. State: Replica: eb083140d82e4d3aa44fecb553f1ff70, State: Running, Role: LEADER
I20260926 01:10:47.405131 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384993574762-392-0/minicluster-data/master-0/wal/instance
I20260926 01:10:47.405156 10169 sys_catalog.cc:564] T 00000000000000000000000000000000 P eb083140d82e4d3aa44fecb553f1ff70 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:10:47.405165 10172 consensus_queue.cc:237] T 00000000000000000000000000000000 P eb083140d82e4d3aa44fecb553f1ff70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb083140d82e4d3aa44fecb553f1ff70" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37905 } }
I20260926 01:10:47.405323 392 external_mini_cluster.cc:1428] Setting key 5983bbbc889c8cfbf1bca5b119a6c85c
I20260926 01:10:47.405966 10173 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb083140d82e4d3aa44fecb553f1ff70 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb083140d82e4d3aa44fecb553f1ff70. Latest consensus state: current_term: 1 leader_uuid: "eb083140d82e4d3aa44fecb553f1ff70" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb083140d82e4d3aa44fecb553f1ff70" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37905 } } }
I20260926 01:10:47.406337 10174 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb083140d82e4d3aa44fecb553f1ff70 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eb083140d82e4d3aa44fecb553f1ff70" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb083140d82e4d3aa44fecb553f1ff70" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37905 } } }
I20260926 01:10:47.406411 10174 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb083140d82e4d3aa44fecb553f1ff70 [sys.catalog]: This master's current role is: LEADER
I20260926 01:10:47.406375 10173 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb083140d82e4d3aa44fecb553f1ff70 [sys.catalog]: This master's current role is: LEADER
W20260926 01:10:47.406967 10189 catalog_manager.cc:1558] T 00000000000000000000000000000000 P eb083140d82e4d3aa44fecb553f1ff70: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 01:10:47.407018 10189 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 01:10:47.407068 10190 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:10:47.407228 10190 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:10:47.408110 10182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:47.406034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43860 (local address 127.0.98.62:37905)
0926 01:10:47.406685 (+ 651us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:47.406695 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:47.406731 (+ 36us) server_negotiation.cc:407] Connection header received
0926 01:10:47.406842 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:47.406847 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:47.406882 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:47.407006 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:47.407653 (+ 647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:10:47.407657 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:10:47.407657 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:10:47.407660 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:10:47.407700 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:10:47.407712 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:10:47.407767 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:47.407839 (+ 72us) server_negotiation.cc:299] Negotiation successful
0926 01:10:47.407972 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":77,"threads_started":1}
I20260926 01:10:47.408928 10190 catalog_manager.cc:1347] Generated new cluster ID: 3d9eb3ef22f34a38a06e13307103aba7
I20260926 01:10:47.408982 10190 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 01:10:47.429698 10190 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:10:47.429947 10190 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:10:47.436879 10190 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eb083140d82e4d3aa44fecb553f1ff70: Generated new TSK 0
I20260926 01:10:47.437134 10190 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 01:10:47.438818 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384993574762-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384993574762-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384993574762-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384993574762-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37905
--builtin_ntp_servers=127.0.98.20:41357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 01:10:47.527264 10193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:10:47.527491 10193 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:10:47.527514 10193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:10:47.528802 10193 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:10:47.528851 10193 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:10:47.528877 10193 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 01:10:47.530344 10193 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384993574762-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384993574762-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384993574762-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37905
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10193
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384993574762-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:10:47.530601 10193 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:10:47.530898 10193 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:10:47.535763 10198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:10:47.535759 10199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:10:47.535770 10201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:10:47.536223 10193 server_base.cc:1034] running on GCE node
I20260926 01:10:47.536409 10193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:10:47.536641 10193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:10:47.537871 10193 hybrid_clock.cc:648] HybridClock initialized: now 1790385047537857 us; error 30 us; skew 500 ppm
I20260926 01:10:47.539402 10193 webserver.cc:460] Webserver started at http://127.0.98.1:46795/ using document root <none> and password file <none>
I20260926 01:10:47.539659 10193 fs_manager.cc:359] Metadata directory not provided
I20260926 01:10:47.539709 10193 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:10:47.539840 10193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:10:47.541266 10193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384993574762-392-0/minicluster-data/ts-0/data/instance:
uuid: "e30de978c4154d378dcdbfc8f071557b"
format_stamp: "Formatted at 2026-09-26 01:10:47 on dist-test-slave-7b22"
server_key: "e8b9d4bb8d5c66a1b6f263b98eb8a0d5"
server_key_iv: "f0505f13fe1aabc5dedf29fd705d9d12"
server_key_version: "encryptionkey@0"
I20260926 01:10:47.542016 10193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384993574762-392-0/minicluster-data/ts-0/wal/instance:
uuid: "e30de978c4154d378dcdbfc8f071557b"
format_stamp: "Formatted at 2026-09-26 01:10:47 on dist-test-slave-7b22"
server_key: "e8b9d4bb8d5c66a1b6f263b98eb8a0d5"
server_key_iv: "f0505f13fe1aabc5dedf29fd705d9d12"
server_key_version: "encryptionkey@0"
I20260926 01:10:47.543455 10193 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:10:47.544375 10207 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:47.544601 10193 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:10:47.544694 10193 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384993574762-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384993574762-392-0/minicluster-data/ts-0/wal
uuid: "e30de978c4154d378dcdbfc8f071557b"
format_stamp: "Formatted at 2026-09-26 01:10:47 on dist-test-slave-7b22"
server_key: "e8b9d4bb8d5c66a1b6f263b98eb8a0d5"
server_key_iv: "f0505f13fe1aabc5dedf29fd705d9d12"
server_key_version: "encryptionkey@0"
I20260926 01:10:47.544762 10193 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384993574762-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384993574762-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384993574762-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:10:47.548198 10193 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:10:47.548473 10193 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:10:47.548600 10193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:10:47.548810 10193 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:10:47.549103 10193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:10:47.549134 10193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:47.549161 10193 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:10:47.549185 10193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:47.554678 10193 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42209
I20260926 01:10:47.554731 10320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42209 every 8 connection(s)
I20260926 01:10:47.555053 10193 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384993574762-392-0/minicluster-data/ts-0/data/info.pb
I20260926 01:10:47.558262 10182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:47.555767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48605 (local address 127.0.98.62:37905)
0926 01:10:47.555831 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:47.555834 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:47.556093 (+ 259us) server_negotiation.cc:407] Connection header received
0926 01:10:47.556581 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:47.556583 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:47.556589 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:47.556644 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:47.557800 (+ 1156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:10:47.557801 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:10:47.557802 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:10:47.557807 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:10:47.557847 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:10:47.557853 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:10:47.557990 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:47.558186 (+ 196us) server_negotiation.cc:299] Negotiation successful
0926 01:10:47.558203 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 01:10:47.558418 10323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:47.555752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37905 (local address 127.0.98.1:48605)
0926 01:10:47.555927 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:47.555964 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:47.556474 (+ 510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:47.556707 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:47.556711 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:47.556719 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:10:47.557621 (+ 902us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 01:10:47.557621 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:10:47.557651 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 01:10:47.557682 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:10:47.558064 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:10:47.558070 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:10:47.558073 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 01:10:47.558153 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 01:10:47.558210 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":82,"threads_started":1}
I20260926 01:10:47.559099 10321 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37905
I20260926 01:10:47.559198 10321 heartbeater.cc:464] Registering TS with master...
I20260926 01:10:47.559342 10321 heartbeater.cc:511] Master 127.0.98.62:37905 requested a full tablet report, sending...
I20260926 01:10:47.559852 10133 ts_manager.cc:194] Registered new tserver with Master: e30de978c4154d378dcdbfc8f071557b (127.0.98.1:42209)
I20260926 01:10:47.564540 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 10193
I20260926 01:10:47.564634 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384993574762-392-0/minicluster-data/ts-0/wal/instance
I20260926 01:10:47.564811 392 external_mini_cluster.cc:1428] Setting key c293fe91a7764c8b9cd84993a4928aff
I20260926 01:10:47.565882 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384993574762-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384993574762-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384993574762-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384993574762-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37905
--builtin_ntp_servers=127.0.98.20:41357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 01:10:47.646732 10324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:10:47.646943 10324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:10:47.646965 10324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:10:47.648231 10324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:10:47.648275 10324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:10:47.648300 10324 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 01:10:47.649709 10324 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384993574762-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384993574762-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384993574762-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37905
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10324
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384993574762-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:10:47.649976 10324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:10:47.650300 10324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:10:47.655511 10329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:10:47.655542 10324 server_base.cc:1034] running on GCE node
W20260926 01:10:47.655541 10332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:10:47.655522 10330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:10:47.655870 10324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:10:47.656118 10324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:10:47.657270 10324 hybrid_clock.cc:648] HybridClock initialized: now 1790385047657251 us; error 29 us; skew 500 ppm
I20260926 01:10:47.658600 10324 webserver.cc:460] Webserver started at http://127.0.98.2:39377/ using document root <none> and password file <none>
I20260926 01:10:47.658838 10324 fs_manager.cc:359] Metadata directory not provided
I20260926 01:10:47.658887 10324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:10:47.659019 10324 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:10:47.660089 10324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384993574762-392-0/minicluster-data/ts-1/data/instance:
uuid: "acf8c65428ce42c49df2a719adf2353a"
format_stamp: "Formatted at 2026-09-26 01:10:47 on dist-test-slave-7b22"
server_key: "ba87902c4701cf71e7fa985795d1b480"
server_key_iv: "84e63c9682d921327477f80fedd712d4"
server_key_version: "encryptionkey@0"
I20260926 01:10:47.660419 10324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384993574762-392-0/minicluster-data/ts-1/wal/instance:
uuid: "acf8c65428ce42c49df2a719adf2353a"
format_stamp: "Formatted at 2026-09-26 01:10:47 on dist-test-slave-7b22"
server_key: "ba87902c4701cf71e7fa985795d1b480"
server_key_iv: "84e63c9682d921327477f80fedd712d4"
server_key_version: "encryptionkey@0"
I20260926 01:10:47.661713 10324 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 01:10:47.662572 10338 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:47.662740 10324 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 01:10:47.662830 10324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384993574762-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384993574762-392-0/minicluster-data/ts-1/wal
uuid: "acf8c65428ce42c49df2a719adf2353a"
format_stamp: "Formatted at 2026-09-26 01:10:47 on dist-test-slave-7b22"
server_key: "ba87902c4701cf71e7fa985795d1b480"
server_key_iv: "84e63c9682d921327477f80fedd712d4"
server_key_version: "encryptionkey@0"
I20260926 01:10:47.662914 10324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384993574762-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384993574762-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384993574762-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:10:47.666266 10324 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:10:47.666508 10324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:10:47.666603 10324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:10:47.666754 10324 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:10:47.666983 10324 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:10:47.667011 10324 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:47.667030 10324 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:10:47.667042 10324 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:47.672477 10324 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33851
I20260926 01:10:47.672549 10451 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33851 every 8 connection(s)
I20260926 01:10:47.672838 10324 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384993574762-392-0/minicluster-data/ts-1/data/info.pb
I20260926 01:10:47.675803 10182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:47.673633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55863 (local address 127.0.98.62:37905)
0926 01:10:47.673687 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:47.673689 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:47.673889 (+ 200us) server_negotiation.cc:407] Connection header received
0926 01:10:47.674400 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:47.674400 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:47.674405 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:47.674443 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:47.675574 (+ 1131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:10:47.675575 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:10:47.675575 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:10:47.675581 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:10:47.675601 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:10:47.675606 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:10:47.675675 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:47.675724 (+ 49us) server_negotiation.cc:299] Negotiation successful
0926 01:10:47.675736 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 01:10:47.675961 10454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:47.673578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37905 (local address 127.0.98.2:55863)
0926 01:10:47.673758 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:47.673797 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:47.674297 (+ 500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:47.674504 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:47.674507 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:47.674515 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:10:47.675406 (+ 891us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 01:10:47.675407 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:10:47.675435 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 01:10:47.675465 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:10:47.675676 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:10:47.675678 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:10:47.675680 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 01:10:47.675730 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 01:10:47.675776 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":65,"threads_started":1}
I20260926 01:10:47.676378 10452 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37905
I20260926 01:10:47.676463 10452 heartbeater.cc:464] Registering TS with master...
I20260926 01:10:47.676600 10452 heartbeater.cc:511] Master 127.0.98.62:37905 requested a full tablet report, sending...
I20260926 01:10:47.676963 10133 ts_manager.cc:194] Registered new tserver with Master: acf8c65428ce42c49df2a719adf2353a (127.0.98.2:33851)
I20260926 01:10:47.680757 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 10324
I20260926 01:10:47.680851 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384993574762-392-0/minicluster-data/ts-1/wal/instance
I20260926 01:10:47.680992 392 external_mini_cluster.cc:1428] Setting key 90adba066d2be55bcdd0b27dbffb9eaa
I20260926 01:10:47.681981 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384993574762-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384993574762-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384993574762-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384993574762-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37905
--builtin_ntp_servers=127.0.98.20:41357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 01:10:47.765179 10455 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:10:47.765461 10455 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:10:47.765496 10455 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:10:47.767066 10455 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:10:47.767127 10455 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:10:47.767165 10455 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 01:10:47.768713 10455 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384993574762-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384993574762-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384993574762-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37905
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10455
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384993574762-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:10:47.768994 10455 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:10:47.769297 10455 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 01:10:47.774605 10455 server_base.cc:1034] running on GCE node
W20260926 01:10:47.774569 10461 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:10:47.774535 10460 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:10:47.774535 10463 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:10:47.775135 10455 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:10:47.775398 10455 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:10:47.776585 10455 hybrid_clock.cc:648] HybridClock initialized: now 1790385047776571 us; error 42 us; skew 500 ppm
I20260926 01:10:47.778033 10455 webserver.cc:460] Webserver started at http://127.0.98.3:33161/ using document root <none> and password file <none>
I20260926 01:10:47.778275 10455 fs_manager.cc:359] Metadata directory not provided
I20260926 01:10:47.778323 10455 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:10:47.778446 10455 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:10:47.779577 10455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384993574762-392-0/minicluster-data/ts-2/data/instance:
uuid: "3f138a5bc36540399dc107958560b830"
format_stamp: "Formatted at 2026-09-26 01:10:47 on dist-test-slave-7b22"
server_key: "16ce6e31ae953127a6eb11260313da97"
server_key_iv: "eb2ca53e84f84aeb5035f822673f7580"
server_key_version: "encryptionkey@0"
I20260926 01:10:47.779907 10455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384993574762-392-0/minicluster-data/ts-2/wal/instance:
uuid: "3f138a5bc36540399dc107958560b830"
format_stamp: "Formatted at 2026-09-26 01:10:47 on dist-test-slave-7b22"
server_key: "16ce6e31ae953127a6eb11260313da97"
server_key_iv: "eb2ca53e84f84aeb5035f822673f7580"
server_key_version: "encryptionkey@0"
I20260926 01:10:47.781487 10455 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 01:10:47.782401 10469 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:47.782589 10455 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 01:10:47.782680 10455 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384993574762-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384993574762-392-0/minicluster-data/ts-2/wal
uuid: "3f138a5bc36540399dc107958560b830"
format_stamp: "Formatted at 2026-09-26 01:10:47 on dist-test-slave-7b22"
server_key: "16ce6e31ae953127a6eb11260313da97"
server_key_iv: "eb2ca53e84f84aeb5035f822673f7580"
server_key_version: "encryptionkey@0"
I20260926 01:10:47.782752 10455 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384993574762-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384993574762-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384993574762-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:10:47.786609 10455 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:10:47.786872 10455 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:10:47.786988 10455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:10:47.787174 10455 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:10:47.787472 10455 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:10:47.787504 10455 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:47.787532 10455 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:10:47.787556 10455 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:47.793180 10455 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36903
I20260926 01:10:47.793264 10582 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36903 every 8 connection(s)
I20260926 01:10:47.793565 10455 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384993574762-392-0/minicluster-data/ts-2/data/info.pb
I20260926 01:10:47.796595 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 10455
I20260926 01:10:47.796710 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384993574762-392-0/minicluster-data/ts-2/wal/instance
I20260926 01:10:47.796911 392 external_mini_cluster.cc:1428] Setting key 3ce4441b84bf1b0d8cc13b0c2939f0bd
I20260926 01:10:47.796947 10182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:47.794498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40227 (local address 127.0.98.62:37905)
0926 01:10:47.794568 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:47.794570 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:47.794714 (+ 144us) server_negotiation.cc:407] Connection header received
0926 01:10:47.795146 (+ 432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:47.795147 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:47.795156 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:47.795222 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:47.796610 (+ 1388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:10:47.796612 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:10:47.796612 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:10:47.796619 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:10:47.796639 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:10:47.796646 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:10:47.796724 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:47.796851 (+ 127us) server_negotiation.cc:299] Negotiation successful
0926 01:10:47.796872 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 01:10:47.797132 10585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:47.794327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37905 (local address 127.0.98.3:40227)
0926 01:10:47.794540 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:47.794590 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:47.795053 (+ 463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:47.795276 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:47.795280 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:47.795288 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:10:47.796387 (+ 1099us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 01:10:47.796388 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:10:47.796419 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 01:10:47.796450 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:10:47.796753 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:10:47.796760 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:10:47.796763 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 01:10:47.796854 (+ 91us) client_negotiation.cc:240] Negotiation successful
0926 01:10:47.796908 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":68,"threads_started":1}
I20260926 01:10:47.797621 10583 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37905
I20260926 01:10:47.797710 10583 heartbeater.cc:464] Registering TS with master...
I20260926 01:10:47.797892 10583 heartbeater.cc:511] Master 127.0.98.62:37905 requested a full tablet report, sending...
I20260926 01:10:47.798244 10133 ts_manager.cc:194] Registered new tserver with Master: 3f138a5bc36540399dc107958560b830 (127.0.98.3:36903)
I20260926 01:10:47.809453 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 01:10:47.811506 10182 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:10:47.810995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43874 (local address 127.0.98.62:37905)
0926 01:10:47.811083 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:47.811085 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:47.811159 (+ 74us) server_negotiation.cc:407] Connection header received
0926 01:10:47.811235 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:47.811235 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:47.811249 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:47.811297 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:47.811432 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43874: BlockingRecv error: recv got EOF from 127.0.0.1:43874 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 01:10:47.811630 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 10193
I20260926 01:10:47.815690 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 10324
I20260926 01:10:47.819628 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 10455
I20260926 01:10:47.823706 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 10102
2026-09-26T01:10:47Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (559 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384993574762-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 01:10:47 dist-test-slave-7b22 krb5kdc[10593](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 01:10:47 dist-test-slave-7b22 krb5kdc[10593](info): set up 2 sockets
Sep 26 01:10:47 dist-test-slave-7b22 krb5kdc[10593](info): commencing operation
krb5kdc: starting...
W20260926 01:10:48.814564 392 mini_kdc.cc:121] Time spent starting KDC: real 0.972s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384993574762-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384993574762-392-0/krb5kdc/test-user.keytab.
Sep 26 01:10:48 dist-test-slave-7b22 krb5kdc[10593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385048, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T01:10:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:10:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384993574762-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384993574762-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 01:10:48.934082 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384993574762-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384993574762-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384993574762-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33225
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384993574762-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33225
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:10:49.017310 10606 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:10:49.017560 10606 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:10:49.017601 10606 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:10:49.018988 10606 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:10:49.019044 10606 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:10:49.019063 10606 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:10:49.019080 10606 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:10:49.019096 10606 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:10:49.020814 10606 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384993574762-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384993574762-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33225
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384993574762-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33225
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10606
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384993574762-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:10:49.021090 10606 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:10:49.021318 10606 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:10:49.026255 10612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:10:49.026268 10614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:10:49.026268 10611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:10:49.027081 10606 server_base.cc:1034] running on GCE node
I20260926 01:10:49.027354 10606 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:10:49.027669 10606 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:10:49.028842 10606 hybrid_clock.cc:648] HybridClock initialized: now 1790385049028830 us; error 38 us; skew 500 ppm
Sep 26 01:10:49 dist-test-slave-7b22 krb5kdc[10593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385049, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:10:49.031809 10606 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 01:10:49.032262 10606 webserver.cc:460] Webserver started at http://127.0.98.62:35887/ using document root <none> and password file <none>
I20260926 01:10:49.032485 10606 fs_manager.cc:359] Metadata directory not provided
I20260926 01:10:49.032531 10606 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:10:49.032661 10606 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:10:49.033746 10606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384993574762-392-0/minicluster-data/master-0/data/instance:
uuid: "fdf82d80753b409c8ff4dd2e8dcef759"
format_stamp: "Formatted at 2026-09-26 01:10:49 on dist-test-slave-7b22"
server_key: "1ea3e8c1ed57c07a947cb421494a86d2"
server_key_iv: "bfeed51a468b2b8c788f0698ce501c89"
server_key_version: "encryptionkey@0"
I20260926 01:10:49.034132 10606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384993574762-392-0/minicluster-data/master-0/wal/instance:
uuid: "fdf82d80753b409c8ff4dd2e8dcef759"
format_stamp: "Formatted at 2026-09-26 01:10:49 on dist-test-slave-7b22"
server_key: "1ea3e8c1ed57c07a947cb421494a86d2"
server_key_iv: "bfeed51a468b2b8c788f0698ce501c89"
server_key_version: "encryptionkey@0"
I20260926 01:10:49.035641 10606 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:10:49.036528 10621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:49.036749 10606 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:10:49.036864 10606 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384993574762-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384993574762-392-0/minicluster-data/master-0/wal
uuid: "fdf82d80753b409c8ff4dd2e8dcef759"
format_stamp: "Formatted at 2026-09-26 01:10:49 on dist-test-slave-7b22"
server_key: "1ea3e8c1ed57c07a947cb421494a86d2"
server_key_iv: "bfeed51a468b2b8c788f0698ce501c89"
server_key_version: "encryptionkey@0"
I20260926 01:10:49.036953 10606 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384993574762-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384993574762-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384993574762-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:10:49.049481 10606 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:10:49.051625 10606 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:10:49.051760 10606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:10:49.055500 10606 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33225
I20260926 01:10:49.055533 10673 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33225 every 8 connection(s)
I20260926 01:10:49.055863 10606 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
I20260926 01:10:49.056893 10674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:10:49.059303 10674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fdf82d80753b409c8ff4dd2e8dcef759: Bootstrap starting.
I20260926 01:10:49.059402 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 10606
I20260926 01:10:49.059527 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384993574762-392-0/minicluster-data/master-0/wal/instance
I20260926 01:10:49.059713 392 external_mini_cluster.cc:1428] Setting key 3489c2ebc77dea50be569e0b6360acf8
I20260926 01:10:49.059952 10674 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fdf82d80753b409c8ff4dd2e8dcef759: Neither blocks nor log segments found. Creating new log.
I20260926 01:10:49.060266 10674 log.cc:824] T 00000000000000000000000000000000 P fdf82d80753b409c8ff4dd2e8dcef759: Log is configured to *not* fsync() on all Append() calls
I20260926 01:10:49.061012 10674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fdf82d80753b409c8ff4dd2e8dcef759: No bootstrap required, opened a new log
I20260926 01:10:49.063256 10674 raft_consensus.cc:348] T 00000000000000000000000000000000 P fdf82d80753b409c8ff4dd2e8dcef759 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdf82d80753b409c8ff4dd2e8dcef759" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33225 } }
I20260926 01:10:49.063428 10674 raft_consensus.cc:374] T 00000000000000000000000000000000 P fdf82d80753b409c8ff4dd2e8dcef759 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:10:49.063467 10674 raft_consensus.cc:729] T 00000000000000000000000000000000 P fdf82d80753b409c8ff4dd2e8dcef759 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdf82d80753b409c8ff4dd2e8dcef759, State: Initialized, Role: FOLLOWER
I20260926 01:10:49.063606 10674 consensus_queue.cc:260] T 00000000000000000000000000000000 P fdf82d80753b409c8ff4dd2e8dcef759 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdf82d80753b409c8ff4dd2e8dcef759" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33225 } }
I20260926 01:10:49.063688 10674 raft_consensus.cc:388] T 00000000000000000000000000000000 P fdf82d80753b409c8ff4dd2e8dcef759 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:10:49.063719 10674 raft_consensus.cc:482] T 00000000000000000000000000000000 P fdf82d80753b409c8ff4dd2e8dcef759 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:10:49.063767 10674 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fdf82d80753b409c8ff4dd2e8dcef759 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:10:49.064350 10674 raft_consensus.cc:504] T 00000000000000000000000000000000 P fdf82d80753b409c8ff4dd2e8dcef759 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdf82d80753b409c8ff4dd2e8dcef759" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33225 } }
I20260926 01:10:49.064483 10674 leader_election.cc:302] T 00000000000000000000000000000000 P fdf82d80753b409c8ff4dd2e8dcef759 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fdf82d80753b409c8ff4dd2e8dcef759; no voters:
I20260926 01:10:49.064658 10674 leader_election.cc:288] T 00000000000000000000000000000000 P fdf82d80753b409c8ff4dd2e8dcef759 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:10:49.064699 10679 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fdf82d80753b409c8ff4dd2e8dcef759 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:10:49.064849 10679 raft_consensus.cc:686] T 00000000000000000000000000000000 P fdf82d80753b409c8ff4dd2e8dcef759 [term 1 LEADER]: Becoming Leader. State: Replica: fdf82d80753b409c8ff4dd2e8dcef759, State: Running, Role: LEADER
I20260926 01:10:49.064899 10674 sys_catalog.cc:564] T 00000000000000000000000000000000 P fdf82d80753b409c8ff4dd2e8dcef759 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:10:49.065141 10679 consensus_queue.cc:237] T 00000000000000000000000000000000 P fdf82d80753b409c8ff4dd2e8dcef759 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdf82d80753b409c8ff4dd2e8dcef759" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33225 } }
I20260926 01:10:49.066320 10680 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdf82d80753b409c8ff4dd2e8dcef759 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fdf82d80753b409c8ff4dd2e8dcef759" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdf82d80753b409c8ff4dd2e8dcef759" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33225 } } }
I20260926 01:10:49.066397 10680 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdf82d80753b409c8ff4dd2e8dcef759 [sys.catalog]: This master's current role is: LEADER
I20260926 01:10:49.066672 10681 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdf82d80753b409c8ff4dd2e8dcef759 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fdf82d80753b409c8ff4dd2e8dcef759. Latest consensus state: current_term: 1 leader_uuid: "fdf82d80753b409c8ff4dd2e8dcef759" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdf82d80753b409c8ff4dd2e8dcef759" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33225 } } }
W20260926 01:10:49.066825 10694 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fdf82d80753b409c8ff4dd2e8dcef759: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260926 01:10:49.066905 10681 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdf82d80753b409c8ff4dd2e8dcef759 [sys.catalog]: This master's current role is: LEADER
W20260926 01:10:49.066967 10694 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 01:10:49.067062 10695 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 26 01:10:49 dist-test-slave-7b22 krb5kdc[10593](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385048, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:10:49.067225 10695 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:10:49.068702 10695 catalog_manager.cc:1347] Generated new cluster ID: 6289534fce874316ba22cc361e193c27
I20260926 01:10:49.068749 10695 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 01:10:49.072546 10677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:49.060211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36264 (local address 127.0.98.62:33225)
0926 01:10:49.060476 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:49.060486 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:49.060519 (+ 33us) server_negotiation.cc:407] Connection header received
0926 01:10:49.061034 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:49.061043 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:49.061120 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:49.061267 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:49.061869 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:49.063224 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:49.064191 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:49.064799 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:49.064899 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:49.067680 (+ 2781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:10:49.067684 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:10:49.067685 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:10:49.067697 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:10:49.070883 (+ 3186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:49.071278 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:49.071281 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:49.071282 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:49.071391 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:49.071572 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:49.071573 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:49.071573 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:49.071888 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:10:49.072040 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:49.072163 (+ 123us) server_negotiation.cc:299] Negotiation successful
0926 01:10:49.072326 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":100,"threads_started":1}
I20260926 01:10:49.089288 10695 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:10:49.090240 10695 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:10:49.102377 10695 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fdf82d80753b409c8ff4dd2e8dcef759: Generated new TSK 0
I20260926 01:10:49.102617 10695 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384993574762-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384993574762-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 01:10:49.133605 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384993574762-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384993574762-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384993574762-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384993574762-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33225
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384993574762-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:10:49.214643 10700 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:10:49.214851 10700 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:10:49.214874 10700 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:10:49.216002 10700 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:10:49.216037 10700 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:10:49.216063 10700 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 01:10:49.217478 10700 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384993574762-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384993574762-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384993574762-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384993574762-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33225
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10700
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384993574762-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:10:49.217710 10700 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:10:49.217952 10700 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:10:49.222968 10708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:10:49.222977 10706 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:10:49.223066 10705 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:10:49.223125 10700 server_base.cc:1034] running on GCE node
I20260926 01:10:49.223382 10700 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:10:49.223599 10700 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:10:49.224766 10700 hybrid_clock.cc:648] HybridClock initialized: now 1790385049224749 us; error 28 us; skew 500 ppm
Sep 26 01:10:49 dist-test-slave-7b22 krb5kdc[10593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385049, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:10:49.227614 10700 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 01:10:49.228155 10700 webserver.cc:460] Webserver started at http://127.0.98.1:35293/ using document root <none> and password file <none>
I20260926 01:10:49.228386 10700 fs_manager.cc:359] Metadata directory not provided
I20260926 01:10:49.228435 10700 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:10:49.228564 10700 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:10:49.229655 10700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384993574762-392-0/minicluster-data/ts-0/data/instance:
uuid: "26870aaaaf054ecd9a249440f840fe7e"
format_stamp: "Formatted at 2026-09-26 01:10:49 on dist-test-slave-7b22"
server_key: "6122f45ae557fddb47d1106aa3b70bc5"
server_key_iv: "db30e7f79cc7d2682e0ae806657bbae6"
server_key_version: "encryptionkey@0"
I20260926 01:10:49.230031 10700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384993574762-392-0/minicluster-data/ts-0/wal/instance:
uuid: "26870aaaaf054ecd9a249440f840fe7e"
format_stamp: "Formatted at 2026-09-26 01:10:49 on dist-test-slave-7b22"
server_key: "6122f45ae557fddb47d1106aa3b70bc5"
server_key_iv: "db30e7f79cc7d2682e0ae806657bbae6"
server_key_version: "encryptionkey@0"
I20260926 01:10:49.231506 10700 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:10:49.232333 10715 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:49.232532 10700 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:10:49.232649 10700 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384993574762-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384993574762-392-0/minicluster-data/ts-0/wal
uuid: "26870aaaaf054ecd9a249440f840fe7e"
format_stamp: "Formatted at 2026-09-26 01:10:49 on dist-test-slave-7b22"
server_key: "6122f45ae557fddb47d1106aa3b70bc5"
server_key_iv: "db30e7f79cc7d2682e0ae806657bbae6"
server_key_version: "encryptionkey@0"
I20260926 01:10:49.232715 10700 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384993574762-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384993574762-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384993574762-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:10:49.242496 10700 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:10:49.244859 10700 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:10:49.245070 10700 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:10:49.245332 10700 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:10:49.245635 10700 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:10:49.245668 10700 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:49.245698 10700 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:10:49.245721 10700 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:49.251603 10700 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:32777
I20260926 01:10:49.251665 10828 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:32777 every 8 connection(s)
I20260926 01:10:49.251968 10700 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384993574762-392-0/minicluster-data/ts-0/data/info.pb
Sep 26 01:10:49 dist-test-slave-7b22 krb5kdc[10593](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385049, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:10:49.258311 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 10700
I20260926 01:10:49.258411 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384993574762-392-0/minicluster-data/ts-0/wal/instance
I20260926 01:10:49.258597 392 external_mini_cluster.cc:1428] Setting key 4b08de70cf7dd7f16dfb3a40899d21ef
I20260926 01:10:49.261492 10677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:49.252893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49409 (local address 127.0.98.62:33225)
0926 01:10:49.252955 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:49.252957 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:49.253194 (+ 237us) server_negotiation.cc:407] Connection header received
0926 01:10:49.253708 (+ 514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:49.253709 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:49.253721 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:49.253773 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:49.254753 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:49.255268 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:49.256268 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:49.256483 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:49.256541 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:49.258749 (+ 2208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:10:49.258751 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:10:49.258751 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:10:49.258760 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:10:49.260062 (+ 1302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:49.260555 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:49.260556 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:49.260556 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:49.260618 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:49.260876 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:49.260876 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:49.260876 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:49.260974 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:10:49.261020 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:49.261346 (+ 326us) server_negotiation.cc:299] Negotiation successful
0926 01:10:49.261422 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 01:10:49.261727 10831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:49.252869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33225 (local address 127.0.98.1:49409)
0926 01:10:49.253029 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:49.253066 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:49.253610 (+ 544us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:49.253862 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:49.253866 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:49.254449 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:10:49.254654 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:49.254659 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:49.255357 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:49.255358 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:49.256183 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:49.256185 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:49.256555 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:49.256556 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:49.256681 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:49.257221 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:10:49.257229 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:10:49.258659 (+ 1430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:10:49.260144 (+ 1485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:10:49.260152 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:10:49.260155 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:10:49.260466 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:10:49.260676 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:10:49.260677 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:10:49.260678 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:10:49.260820 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:10:49.261025 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:10:49.261030 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:10:49.261215 (+ 185us) client_negotiation.cc:769] Sending connection context
0926 01:10:49.261332 (+ 117us) client_negotiation.cc:240] Negotiation successful
0926 01:10:49.261497 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":77,"threads_started":1}
I20260926 01:10:49.262251 10829 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33225
I20260926 01:10:49.262348 10829 heartbeater.cc:464] Registering TS with master...
I20260926 01:10:49.262544 10829 heartbeater.cc:511] Master 127.0.98.62:33225 requested a full tablet report, sending...
I20260926 01:10:49.263082 10638 ts_manager.cc:194] Registered new tserver with Master: 26870aaaaf054ecd9a249440f840fe7e (127.0.98.1:32777)
I20260926 01:10:49.263700 10638 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:49409
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384993574762-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384993574762-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 01:10:49.284909 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384993574762-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384993574762-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384993574762-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384993574762-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33225
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384993574762-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:10:49.368818 10834 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:10:49.369053 10834 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:10:49.369086 10834 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:10:49.370312 10834 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:10:49.370355 10834 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:10:49.370389 10834 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 01:10:49.371845 10834 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384993574762-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384993574762-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384993574762-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384993574762-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33225
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10834
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384993574762-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:10:49.372087 10834 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:10:49.372299 10834 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:10:49.377564 10840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:10:49.377630 10834 server_base.cc:1034] running on GCE node
W20260926 01:10:49.377578 10839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:10:49.377585 10842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:10:49.378014 10834 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:10:49.378257 10834 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:10:49.379429 10834 hybrid_clock.cc:648] HybridClock initialized: now 1790385049379419 us; error 28 us; skew 500 ppm
Sep 26 01:10:49 dist-test-slave-7b22 krb5kdc[10593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385049, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:10:49.382339 10834 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 01:10:49.382874 10834 webserver.cc:460] Webserver started at http://127.0.98.2:32995/ using document root <none> and password file <none>
I20260926 01:10:49.383088 10834 fs_manager.cc:359] Metadata directory not provided
I20260926 01:10:49.383208 10834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:10:49.383333 10834 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:10:49.384429 10834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384993574762-392-0/minicluster-data/ts-1/data/instance:
uuid: "a695c7b307c249bd897781eaab8da73f"
format_stamp: "Formatted at 2026-09-26 01:10:49 on dist-test-slave-7b22"
server_key: "5b0b92a5406e039ff792a01c30d5e662"
server_key_iv: "93c0bc774143a552eac92b8166982363"
server_key_version: "encryptionkey@0"
I20260926 01:10:49.384763 10834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384993574762-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a695c7b307c249bd897781eaab8da73f"
format_stamp: "Formatted at 2026-09-26 01:10:49 on dist-test-slave-7b22"
server_key: "5b0b92a5406e039ff792a01c30d5e662"
server_key_iv: "93c0bc774143a552eac92b8166982363"
server_key_version: "encryptionkey@0"
I20260926 01:10:49.386315 10834 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 01:10:49.387154 10849 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:49.387337 10834 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:10:49.387415 10834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384993574762-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384993574762-392-0/minicluster-data/ts-1/wal
uuid: "a695c7b307c249bd897781eaab8da73f"
format_stamp: "Formatted at 2026-09-26 01:10:49 on dist-test-slave-7b22"
server_key: "5b0b92a5406e039ff792a01c30d5e662"
server_key_iv: "93c0bc774143a552eac92b8166982363"
server_key_version: "encryptionkey@0"
I20260926 01:10:49.387492 10834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384993574762-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384993574762-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384993574762-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:10:49.402580 10834 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:10:49.404695 10834 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:10:49.404824 10834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:10:49.405063 10834 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:10:49.405357 10834 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:10:49.405390 10834 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:49.405413 10834 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:10:49.405435 10834 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:49.411155 10834 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42293
I20260926 01:10:49.411252 10962 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42293 every 8 connection(s)
I20260926 01:10:49.411531 10834 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384993574762-392-0/minicluster-data/ts-1/data/info.pb
Sep 26 01:10:49 dist-test-slave-7b22 krb5kdc[10593](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385049, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:10:49.420452 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 10834
I20260926 01:10:49.420548 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384993574762-392-0/minicluster-data/ts-1/wal/instance
I20260926 01:10:49.420766 392 external_mini_cluster.cc:1428] Setting key 7121b88f6a4429b5ddb88a361affcc48
I20260926 01:10:49.421113 10677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:49.412243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56049 (local address 127.0.98.62:33225)
0926 01:10:49.412297 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:49.412299 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:49.412473 (+ 174us) server_negotiation.cc:407] Connection header received
0926 01:10:49.412994 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:49.412995 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:49.413007 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:49.413058 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:49.414047 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:49.414571 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:49.415585 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:49.415770 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:49.415826 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:49.418126 (+ 2300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:10:49.418127 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:10:49.418127 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:10:49.418135 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:10:49.419586 (+ 1451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:49.420018 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:49.420019 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:49.420020 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:49.420096 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:49.420376 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:49.420377 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:49.420377 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:49.420517 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:10:49.420600 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:49.420938 (+ 338us) server_negotiation.cc:299] Negotiation successful
0926 01:10:49.421023 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260926 01:10:49.421290 10965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:49.412207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33225 (local address 127.0.98.2:56049)
0926 01:10:49.412349 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:49.412392 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:49.412894 (+ 502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:49.413130 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:49.413134 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:49.413722 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:10:49.413945 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:49.413951 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:49.414666 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:49.414667 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:49.415497 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:49.415500 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:49.415839 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:49.415840 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:49.415963 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:49.416474 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:10:49.416483 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:10:49.418014 (+ 1531us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:10:49.419697 (+ 1683us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:10:49.419704 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:10:49.419706 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:10:49.419947 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:10:49.420153 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:10:49.420154 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:10:49.420154 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:10:49.420286 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:10:49.420594 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:10:49.420601 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:10:49.420777 (+ 176us) client_negotiation.cc:769] Sending connection context
0926 01:10:49.420893 (+ 116us) client_negotiation.cc:240] Negotiation successful
0926 01:10:49.421064 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":63,"threads_started":1}
I20260926 01:10:49.421746 10963 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33225
I20260926 01:10:49.421867 10963 heartbeater.cc:464] Registering TS with master...
I20260926 01:10:49.422048 10963 heartbeater.cc:511] Master 127.0.98.62:33225 requested a full tablet report, sending...
I20260926 01:10:49.422423 10638 ts_manager.cc:194] Registered new tserver with Master: a695c7b307c249bd897781eaab8da73f (127.0.98.2:42293)
I20260926 01:10:49.422885 10638 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:56049
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384993574762-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384993574762-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 01:10:49.445245 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384993574762-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384993574762-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384993574762-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384993574762-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33225
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384993574762-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:10:49.526408 10968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:10:49.526620 10968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:10:49.526644 10968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:10:49.527881 10968 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:10:49.527926 10968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:10:49.527953 10968 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 01:10:49.529464 10968 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384993574762-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384993574762-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384993574762-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384993574762-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33225
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10968
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384993574762-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:10:49.529707 10968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:10:49.529963 10968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:10:49.535198 10976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:10:49.535202 10974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:10:49.535208 10973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:10:49.535594 10968 server_base.cc:1034] running on GCE node
I20260926 01:10:49.535823 10968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:10:49.536108 10968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:10:49.537274 10968 hybrid_clock.cc:648] HybridClock initialized: now 1790385049537253 us; error 38 us; skew 500 ppm
Sep 26 01:10:49 dist-test-slave-7b22 krb5kdc[10593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385049, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:10:49.539932 10968 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 01:10:49.540406 10968 webserver.cc:460] Webserver started at http://127.0.98.3:41473/ using document root <none> and password file <none>
I20260926 01:10:49.540658 10968 fs_manager.cc:359] Metadata directory not provided
I20260926 01:10:49.540709 10968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:10:49.540831 10968 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:10:49.541934 10968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384993574762-392-0/minicluster-data/ts-2/data/instance:
uuid: "3df9a1da0b21408eab766e5acbe57a31"
format_stamp: "Formatted at 2026-09-26 01:10:49 on dist-test-slave-7b22"
server_key: "e073750aa6028be261b927c3d5f6b78b"
server_key_iv: "5dab1f963970d82c0c42c5ecc7b18642"
server_key_version: "encryptionkey@0"
I20260926 01:10:49.542304 10968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384993574762-392-0/minicluster-data/ts-2/wal/instance:
uuid: "3df9a1da0b21408eab766e5acbe57a31"
format_stamp: "Formatted at 2026-09-26 01:10:49 on dist-test-slave-7b22"
server_key: "e073750aa6028be261b927c3d5f6b78b"
server_key_iv: "5dab1f963970d82c0c42c5ecc7b18642"
server_key_version: "encryptionkey@0"
I20260926 01:10:49.543666 10968 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:10:49.544489 10983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:49.544655 10968 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:10:49.544737 10968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384993574762-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384993574762-392-0/minicluster-data/ts-2/wal
uuid: "3df9a1da0b21408eab766e5acbe57a31"
format_stamp: "Formatted at 2026-09-26 01:10:49 on dist-test-slave-7b22"
server_key: "e073750aa6028be261b927c3d5f6b78b"
server_key_iv: "5dab1f963970d82c0c42c5ecc7b18642"
server_key_version: "encryptionkey@0"
I20260926 01:10:49.544808 10968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384993574762-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384993574762-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384993574762-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:10:49.553341 10968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:10:49.555392 10968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:10:49.555531 10968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:10:49.555761 10968 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:10:49.556054 10968 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:10:49.556090 10968 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:49.556118 10968 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:10:49.556142 10968 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:10:49.561746 10968 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42417
I20260926 01:10:49.561791 11096 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42417 every 8 connection(s)
I20260926 01:10:49.562167 10968 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384993574762-392-0/minicluster-data/ts-2/data/info.pb
Sep 26 01:10:49 dist-test-slave-7b22 krb5kdc[10593](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385049, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:10:49.570398 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 10968
I20260926 01:10:49.570501 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384993574762-392-0/minicluster-data/ts-2/wal/instance
I20260926 01:10:49.570679 392 external_mini_cluster.cc:1428] Setting key ca595f208c28a1c84b930de9ffdc9da1
I20260926 01:10:49.571689 10677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:49.562897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41669 (local address 127.0.98.62:33225)
0926 01:10:49.562967 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:49.562969 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:49.563144 (+ 175us) server_negotiation.cc:407] Connection header received
0926 01:10:49.563655 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:49.563656 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:49.563668 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:49.563720 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:49.564693 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:49.565252 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:49.566316 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:49.566529 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:49.566606 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:49.568881 (+ 2275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:10:49.568882 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:10:49.568883 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:10:49.568889 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:10:49.570213 (+ 1324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:49.570653 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:49.570654 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:49.570655 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:49.570708 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:49.571030 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:49.571031 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:49.571031 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:49.571180 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:10:49.571241 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:49.571507 (+ 266us) server_negotiation.cc:299] Negotiation successful
0926 01:10:49.571590 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 01:10:49.571887 11099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:49.562835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33225 (local address 127.0.98.3:41669)
0926 01:10:49.563019 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:49.563065 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:49.563561 (+ 496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:49.563790 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:49.563794 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:49.564403 (+ 609us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:10:49.564600 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:49.564605 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:49.565341 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:49.565342 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:49.566238 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:49.566241 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:49.566680 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:49.566681 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:49.566814 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:49.567339 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:10:49.567348 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:10:49.568789 (+ 1441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:10:49.570319 (+ 1530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:10:49.570326 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:10:49.570328 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:10:49.570585 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:10:49.570806 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:10:49.570806 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:10:49.570807 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:10:49.570957 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:10:49.571230 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:10:49.571236 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:10:49.571383 (+ 147us) client_negotiation.cc:769] Sending connection context
0926 01:10:49.571475 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 01:10:49.571643 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":81,"threads_started":1}
I20260926 01:10:49.572433 11097 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33225
I20260926 01:10:49.572535 11097 heartbeater.cc:464] Registering TS with master...
I20260926 01:10:49.572723 11097 heartbeater.cc:511] Master 127.0.98.62:33225 requested a full tablet report, sending...
I20260926 01:10:49.573071 10638 ts_manager.cc:194] Registered new tserver with Master: 3df9a1da0b21408eab766e5acbe57a31 (127.0.98.3:42417)
I20260926 01:10:49.573479 10638 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:41669
I20260926 01:10:49.582769 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 01:10:49.590353 10677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:49.584251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36276 (local address 127.0.98.62:33225)
0926 01:10:49.584336 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:49.584339 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:49.584403 (+ 64us) server_negotiation.cc:407] Connection header received
0926 01:10:49.584471 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:49.584472 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:49.584483 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:49.584553 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:10:49.585164 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:49.585667 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:49.586452 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:49.586637 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:49.586692 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:49.587866 (+ 1174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:10:49.587867 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:10:49.587867 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:10:49.587875 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:10:49.589174 (+ 1299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:49.589501 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:49.589502 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:49.589502 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:49.589556 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:10:49.589725 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:10:49.589726 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:10:49.589726 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:10:49.589936 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:10:49.590029 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:49.590148 (+ 119us) server_negotiation.cc:299] Negotiation successful
0926 01:10:49.590232 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 01:10:49.591858 10638 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36276:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:10:49.593427 10638 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:10:49.600199 11109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:49.595960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44478 (local address 127.0.98.2:42293)
0926 01:10:49.596135 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:49.596139 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:49.596184 (+ 45us) server_negotiation.cc:407] Connection header received
0926 01:10:49.596256 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:49.596258 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:49.596272 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:49.596371 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:10:49.596491 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:49.597422 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:49.599067 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:49.599736 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:49.599801 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:49.599864 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:49.600052 (+ 188us) server_negotiation.cc:299] Negotiation successful
0926 01:10:49.600100 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":89,"threads_started":1}
I20260926 01:10:49.600208 11107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:49.595967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42417 (local address 127.0.0.1:33858)
0926 01:10:49.596298 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:49.596309 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:49.596351 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:49.596709 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:49.596710 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:49.596712 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:10:49.596756 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:49.596758 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:49.598027 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:49.598028 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:49.598905 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:49.598908 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:49.599824 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:49.599825 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:49.599923 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:49.599949 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 01:10:49.600029 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 01:10:49.600147 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"mutex_wait_us":77,"thread_start_us":51,"threads_started":1}
I20260926 01:10:49.600530 11110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:49.596145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33858 (local address 127.0.98.3:42417)
0926 01:10:49.596490 (+ 345us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:49.596494 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:49.596504 (+ 10us) server_negotiation.cc:407] Connection header received
0926 01:10:49.596550 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:49.596556 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:49.596570 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:49.596636 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:10:49.596843 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:49.597942 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:49.598979 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:49.599723 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:49.599801 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:49.599864 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:49.600401 (+ 537us) server_negotiation.cc:299] Negotiation successful
0926 01:10:49.600449 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":79,"threads_started":1}
I20260926 01:10:49.600201 11108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:49.595844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42293 (local address 127.0.0.1:44478)
0926 01:10:49.596128 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:49.596138 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:49.596193 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:49.596375 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:49.596376 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:49.596378 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:10:49.596418 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:49.596420 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:49.597554 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:49.597555 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:49.598993 (+ 1438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:49.598994 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:49.599823 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:49.599824 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:49.599923 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:49.599949 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 01:10:49.600029 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 01:10:49.600103 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"mutex_wait_us":96,"thread_start_us":39,"threads_started":1}
I20260926 01:10:49.601020 11105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:49.595660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32777 (local address 127.0.0.1:58652)
0926 01:10:49.595792 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:49.595830 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:49.595895 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:49.596209 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:49.596212 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:49.596215 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:10:49.596295 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:49.596297 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:49.597739 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:49.597740 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:49.598991 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:49.598994 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:49.600784 (+ 1790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:49.600786 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:49.600847 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:49.600852 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 01:10:49.600897 (+ 45us) client_negotiation.cc:240] Negotiation successful
0926 01:10:49.600943 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"mutex_wait_us":9,"thread_start_us":55,"threads_started":1}
I20260926 01:10:49.601246 11106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:49.595767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58652 (local address 127.0.98.1:32777)
0926 01:10:49.595991 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:49.595996 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:49.596007 (+ 11us) server_negotiation.cc:407] Connection header received
0926 01:10:49.596046 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:49.596049 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:49.596064 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:49.596363 (+ 299us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:10:49.596513 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:49.597635 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:49.599903 (+ 2268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:49.600566 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:49.600667 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:49.600734 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:49.601119 (+ 385us) server_negotiation.cc:299] Negotiation successful
0926 01:10:49.601167 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":75,"threads_started":1}
I20260926 01:10:49.601344 11031 tablet_service.cc:1467] Processing CreateTablet for tablet 54febe924d6143b4aa4c3fd7db942ba4 (DEFAULT_TABLE table=test-table [id=b0719f991ae04387bcfa3038ea6dad83]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:10:49.601747 11031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54febe924d6143b4aa4c3fd7db942ba4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:10:49.601882 10763 tablet_service.cc:1467] Processing CreateTablet for tablet 54febe924d6143b4aa4c3fd7db942ba4 (DEFAULT_TABLE table=test-table [id=b0719f991ae04387bcfa3038ea6dad83]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:10:49.602209 10763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54febe924d6143b4aa4c3fd7db942ba4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:10:49.605281 11112 tablet_bootstrap.cc:492] T 54febe924d6143b4aa4c3fd7db942ba4 P 26870aaaaf054ecd9a249440f840fe7e: Bootstrap starting.
I20260926 01:10:49.605281 11111 tablet_bootstrap.cc:492] T 54febe924d6143b4aa4c3fd7db942ba4 P 3df9a1da0b21408eab766e5acbe57a31: Bootstrap starting.
I20260926 01:10:49.605718 10897 tablet_service.cc:1467] Processing CreateTablet for tablet 54febe924d6143b4aa4c3fd7db942ba4 (DEFAULT_TABLE table=test-table [id=b0719f991ae04387bcfa3038ea6dad83]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:10:49.606034 11112 tablet_bootstrap.cc:654] T 54febe924d6143b4aa4c3fd7db942ba4 P 26870aaaaf054ecd9a249440f840fe7e: Neither blocks nor log segments found. Creating new log.
I20260926 01:10:49.606036 11111 tablet_bootstrap.cc:654] T 54febe924d6143b4aa4c3fd7db942ba4 P 3df9a1da0b21408eab766e5acbe57a31: Neither blocks nor log segments found. Creating new log.
I20260926 01:10:49.606050 10897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54febe924d6143b4aa4c3fd7db942ba4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:10:49.606631 11111 log.cc:824] T 54febe924d6143b4aa4c3fd7db942ba4 P 3df9a1da0b21408eab766e5acbe57a31: Log is configured to *not* fsync() on all Append() calls
I20260926 01:10:49.606631 11112 log.cc:824] T 54febe924d6143b4aa4c3fd7db942ba4 P 26870aaaaf054ecd9a249440f840fe7e: Log is configured to *not* fsync() on all Append() calls
I20260926 01:10:49.607517 11111 tablet_bootstrap.cc:492] T 54febe924d6143b4aa4c3fd7db942ba4 P 3df9a1da0b21408eab766e5acbe57a31: No bootstrap required, opened a new log
I20260926 01:10:49.607591 11111 ts_tablet_manager.cc:1397] T 54febe924d6143b4aa4c3fd7db942ba4 P 3df9a1da0b21408eab766e5acbe57a31: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 01:10:49.607596 11112 tablet_bootstrap.cc:492] T 54febe924d6143b4aa4c3fd7db942ba4 P 26870aaaaf054ecd9a249440f840fe7e: No bootstrap required, opened a new log
I20260926 01:10:49.607657 11112 ts_tablet_manager.cc:1397] T 54febe924d6143b4aa4c3fd7db942ba4 P 26870aaaaf054ecd9a249440f840fe7e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 01:10:49.609023 11111 raft_consensus.cc:348] T 54febe924d6143b4aa4c3fd7db942ba4 P 3df9a1da0b21408eab766e5acbe57a31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26870aaaaf054ecd9a249440f840fe7e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32777 } } peers { permanent_uuid: "a695c7b307c249bd897781eaab8da73f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42293 } } peers { permanent_uuid: "3df9a1da0b21408eab766e5acbe57a31" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42417 } }
I20260926 01:10:49.609181 11111 raft_consensus.cc:374] T 54febe924d6143b4aa4c3fd7db942ba4 P 3df9a1da0b21408eab766e5acbe57a31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:10:49.609213 11111 raft_consensus.cc:729] T 54febe924d6143b4aa4c3fd7db942ba4 P 3df9a1da0b21408eab766e5acbe57a31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3df9a1da0b21408eab766e5acbe57a31, State: Initialized, Role: FOLLOWER
I20260926 01:10:49.609333 11111 consensus_queue.cc:260] T 54febe924d6143b4aa4c3fd7db942ba4 P 3df9a1da0b21408eab766e5acbe57a31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26870aaaaf054ecd9a249440f840fe7e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32777 } } peers { permanent_uuid: "a695c7b307c249bd897781eaab8da73f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42293 } } peers { permanent_uuid: "3df9a1da0b21408eab766e5acbe57a31" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42417 } }
I20260926 01:10:49.609627 11111 ts_tablet_manager.cc:1428] T 54febe924d6143b4aa4c3fd7db942ba4 P 3df9a1da0b21408eab766e5acbe57a31: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 01:10:49.609632 11097 heartbeater.cc:503] Master 127.0.98.62:33225 was elected leader, sending a full tablet report...
I20260926 01:10:49.610009 11115 tablet_bootstrap.cc:492] T 54febe924d6143b4aa4c3fd7db942ba4 P a695c7b307c249bd897781eaab8da73f: Bootstrap starting.
I20260926 01:10:49.610114 11112 raft_consensus.cc:348] T 54febe924d6143b4aa4c3fd7db942ba4 P 26870aaaaf054ecd9a249440f840fe7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26870aaaaf054ecd9a249440f840fe7e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32777 } } peers { permanent_uuid: "a695c7b307c249bd897781eaab8da73f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42293 } } peers { permanent_uuid: "3df9a1da0b21408eab766e5acbe57a31" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42417 } }
I20260926 01:10:49.610296 11112 raft_consensus.cc:374] T 54febe924d6143b4aa4c3fd7db942ba4 P 26870aaaaf054ecd9a249440f840fe7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:10:49.610339 11112 raft_consensus.cc:729] T 54febe924d6143b4aa4c3fd7db942ba4 P 26870aaaaf054ecd9a249440f840fe7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26870aaaaf054ecd9a249440f840fe7e, State: Initialized, Role: FOLLOWER
I20260926 01:10:49.610466 11112 consensus_queue.cc:260] T 54febe924d6143b4aa4c3fd7db942ba4 P 26870aaaaf054ecd9a249440f840fe7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26870aaaaf054ecd9a249440f840fe7e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32777 } } peers { permanent_uuid: "a695c7b307c249bd897781eaab8da73f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42293 } } peers { permanent_uuid: "3df9a1da0b21408eab766e5acbe57a31" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42417 } }
I20260926 01:10:49.610711 11115 tablet_bootstrap.cc:654] T 54febe924d6143b4aa4c3fd7db942ba4 P a695c7b307c249bd897781eaab8da73f: Neither blocks nor log segments found. Creating new log.
I20260926 01:10:49.610781 11112 ts_tablet_manager.cc:1428] T 54febe924d6143b4aa4c3fd7db942ba4 P 26870aaaaf054ecd9a249440f840fe7e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 01:10:49.610849 10829 heartbeater.cc:503] Master 127.0.98.62:33225 was elected leader, sending a full tablet report...
I20260926 01:10:49.611258 11115 log.cc:824] T 54febe924d6143b4aa4c3fd7db942ba4 P a695c7b307c249bd897781eaab8da73f: Log is configured to *not* fsync() on all Append() calls
I20260926 01:10:49.612118 11115 tablet_bootstrap.cc:492] T 54febe924d6143b4aa4c3fd7db942ba4 P a695c7b307c249bd897781eaab8da73f: No bootstrap required, opened a new log
I20260926 01:10:49.612284 11115 ts_tablet_manager.cc:1397] T 54febe924d6143b4aa4c3fd7db942ba4 P a695c7b307c249bd897781eaab8da73f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 01:10:49.613695 11115 raft_consensus.cc:348] T 54febe924d6143b4aa4c3fd7db942ba4 P a695c7b307c249bd897781eaab8da73f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26870aaaaf054ecd9a249440f840fe7e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32777 } } peers { permanent_uuid: "a695c7b307c249bd897781eaab8da73f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42293 } } peers { permanent_uuid: "3df9a1da0b21408eab766e5acbe57a31" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42417 } }
I20260926 01:10:49.613893 11115 raft_consensus.cc:374] T 54febe924d6143b4aa4c3fd7db942ba4 P a695c7b307c249bd897781eaab8da73f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:10:49.613933 11115 raft_consensus.cc:729] T 54febe924d6143b4aa4c3fd7db942ba4 P a695c7b307c249bd897781eaab8da73f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a695c7b307c249bd897781eaab8da73f, State: Initialized, Role: FOLLOWER
I20260926 01:10:49.614074 11115 consensus_queue.cc:260] T 54febe924d6143b4aa4c3fd7db942ba4 P a695c7b307c249bd897781eaab8da73f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26870aaaaf054ecd9a249440f840fe7e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32777 } } peers { permanent_uuid: "a695c7b307c249bd897781eaab8da73f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42293 } } peers { permanent_uuid: "3df9a1da0b21408eab766e5acbe57a31" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42417 } }
I20260926 01:10:49.614303 11115 ts_tablet_manager.cc:1428] T 54febe924d6143b4aa4c3fd7db942ba4 P a695c7b307c249bd897781eaab8da73f: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 01:10:49.614367 10963 heartbeater.cc:503] Master 127.0.98.62:33225 was elected leader, sending a full tablet report...
W20260926 01:10:49.662120 10964 tablet.cc:2369] T 54febe924d6143b4aa4c3fd7db942ba4 P a695c7b307c249bd897781eaab8da73f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 01:10:49.752965 10830 tablet.cc:2369] T 54febe924d6143b4aa4c3fd7db942ba4 P 26870aaaaf054ecd9a249440f840fe7e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 01:10:49.812742 11098 tablet.cc:2369] T 54febe924d6143b4aa4c3fd7db942ba4 P 3df9a1da0b21408eab766e5acbe57a31: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 01:10:49.853219 11117 raft_consensus.cc:482] T 54febe924d6143b4aa4c3fd7db942ba4 P 26870aaaaf054ecd9a249440f840fe7e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:10:49.853367 11117 raft_consensus.cc:504] T 54febe924d6143b4aa4c3fd7db942ba4 P 26870aaaaf054ecd9a249440f840fe7e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26870aaaaf054ecd9a249440f840fe7e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32777 } } peers { permanent_uuid: "a695c7b307c249bd897781eaab8da73f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42293 } } peers { permanent_uuid: "3df9a1da0b21408eab766e5acbe57a31" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42417 } }
I20260926 01:10:49.853754 11117 leader_election.cc:288] T 54febe924d6143b4aa4c3fd7db942ba4 P 26870aaaaf054ecd9a249440f840fe7e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a695c7b307c249bd897781eaab8da73f (127.0.98.2:42293), 3df9a1da0b21408eab766e5acbe57a31 (127.0.98.3:42417)
I20260926 01:10:49.856706 11109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:49.854200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49899 (local address 127.0.98.2:42293)
0926 01:10:49.854262 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:49.854265 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:49.854277 (+ 12us) server_negotiation.cc:407] Connection header received
0926 01:10:49.854366 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:49.854368 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:49.854379 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:49.854425 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:10:49.854586 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:49.855059 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:49.856027 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:49.856390 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:49.856449 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:49.856461 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:49.856599 (+ 138us) server_negotiation.cc:299] Negotiation successful
0926 01:10:49.856638 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 01:10:49.856748 11120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:49.854065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42293 (local address 127.0.98.1:49899)
0926 01:10:49.854205 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:49.854217 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:49.854275 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:49.854438 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:49.854439 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:49.854441 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:10:49.854523 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:49.854525 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:49.855140 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:49.855141 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:49.855935 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:49.855938 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:49.856471 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:49.856472 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:49.856537 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:49.856540 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 01:10:49.856584 (+ 44us) client_negotiation.cc:240] Negotiation successful
0926 01:10:49.856629 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":72,"threads_started":1}
I20260926 01:10:49.856763 11110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:49.854319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48239 (local address 127.0.98.3:42417)
0926 01:10:49.854359 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:49.854361 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:49.854421 (+ 60us) server_negotiation.cc:407] Connection header received
0926 01:10:49.854483 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:49.854484 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:49.854495 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:49.854545 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:10:49.854697 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:49.855156 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:49.856074 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:49.856443 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:49.856502 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:49.856512 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:49.856657 (+ 145us) server_negotiation.cc:299] Negotiation successful
0926 01:10:49.856698 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 01:10:49.856812 11121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:49.854240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42417 (local address 127.0.98.1:48239)
0926 01:10:49.854337 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:49.854347 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 01:10:49.854428 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:10:49.854556 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:10:49.854557 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:10:49.854558 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:10:49.854632 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:49.854633 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:49.855230 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:49.855231 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:49.855990 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:10:49.855992 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:10:49.856515 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:10:49.856517 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:10:49.856595 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:49.856598 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 01:10:49.856642 (+ 44us) client_negotiation.cc:240] Negotiation successful
0926 01:10:49.856683 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83,"thread_start_us":49,"threads_started":1}
I20260926 01:10:49.857091 11051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54febe924d6143b4aa4c3fd7db942ba4" candidate_uuid: "26870aaaaf054ecd9a249440f840fe7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3df9a1da0b21408eab766e5acbe57a31" is_pre_election: true
I20260926 01:10:49.857152 10917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54febe924d6143b4aa4c3fd7db942ba4" candidate_uuid: "26870aaaaf054ecd9a249440f840fe7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a695c7b307c249bd897781eaab8da73f" is_pre_election: true
I20260926 01:10:49.857252 11051 raft_consensus.cc:2454] T 54febe924d6143b4aa4c3fd7db942ba4 P 3df9a1da0b21408eab766e5acbe57a31 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26870aaaaf054ecd9a249440f840fe7e in term 0.
I20260926 01:10:49.857290 10917 raft_consensus.cc:2454] T 54febe924d6143b4aa4c3fd7db942ba4 P a695c7b307c249bd897781eaab8da73f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26870aaaaf054ecd9a249440f840fe7e in term 0.
I20260926 01:10:49.857486 10718 leader_election.cc:302] T 54febe924d6143b4aa4c3fd7db942ba4 P 26870aaaaf054ecd9a249440f840fe7e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26870aaaaf054ecd9a249440f840fe7e, 3df9a1da0b21408eab766e5acbe57a31; no voters:
I20260926 01:10:49.857620 11117 raft_consensus.cc:2781] T 54febe924d6143b4aa4c3fd7db942ba4 P 26870aaaaf054ecd9a249440f840fe7e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:10:49.857699 11117 raft_consensus.cc:482] T 54febe924d6143b4aa4c3fd7db942ba4 P 26870aaaaf054ecd9a249440f840fe7e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:10:49.857715 11117 raft_consensus.cc:3037] T 54febe924d6143b4aa4c3fd7db942ba4 P 26870aaaaf054ecd9a249440f840fe7e [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:10:49.858469 11117 raft_consensus.cc:504] T 54febe924d6143b4aa4c3fd7db942ba4 P 26870aaaaf054ecd9a249440f840fe7e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26870aaaaf054ecd9a249440f840fe7e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32777 } } peers { permanent_uuid: "a695c7b307c249bd897781eaab8da73f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42293 } } peers { permanent_uuid: "3df9a1da0b21408eab766e5acbe57a31" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42417 } }
I20260926 01:10:49.858584 11117 leader_election.cc:288] T 54febe924d6143b4aa4c3fd7db942ba4 P 26870aaaaf054ecd9a249440f840fe7e [CANDIDATE]: Term 1 election: Requested vote from peers a695c7b307c249bd897781eaab8da73f (127.0.98.2:42293), 3df9a1da0b21408eab766e5acbe57a31 (127.0.98.3:42417)
I20260926 01:10:49.858779 10917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54febe924d6143b4aa4c3fd7db942ba4" candidate_uuid: "26870aaaaf054ecd9a249440f840fe7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a695c7b307c249bd897781eaab8da73f"
I20260926 01:10:49.858825 11051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54febe924d6143b4aa4c3fd7db942ba4" candidate_uuid: "26870aaaaf054ecd9a249440f840fe7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3df9a1da0b21408eab766e5acbe57a31"
I20260926 01:10:49.858865 10917 raft_consensus.cc:3037] T 54febe924d6143b4aa4c3fd7db942ba4 P a695c7b307c249bd897781eaab8da73f [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:10:49.858889 11051 raft_consensus.cc:3037] T 54febe924d6143b4aa4c3fd7db942ba4 P 3df9a1da0b21408eab766e5acbe57a31 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:10:49.859598 11051 raft_consensus.cc:2454] T 54febe924d6143b4aa4c3fd7db942ba4 P 3df9a1da0b21408eab766e5acbe57a31 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26870aaaaf054ecd9a249440f840fe7e in term 1.
I20260926 01:10:49.859637 10917 raft_consensus.cc:2454] T 54febe924d6143b4aa4c3fd7db942ba4 P a695c7b307c249bd897781eaab8da73f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26870aaaaf054ecd9a249440f840fe7e in term 1.
I20260926 01:10:49.859757 10718 leader_election.cc:302] T 54febe924d6143b4aa4c3fd7db942ba4 P 26870aaaaf054ecd9a249440f840fe7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26870aaaaf054ecd9a249440f840fe7e, 3df9a1da0b21408eab766e5acbe57a31; no voters:
I20260926 01:10:49.859854 11117 raft_consensus.cc:2781] T 54febe924d6143b4aa4c3fd7db942ba4 P 26870aaaaf054ecd9a249440f840fe7e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:10:49.860018 11117 raft_consensus.cc:686] T 54febe924d6143b4aa4c3fd7db942ba4 P 26870aaaaf054ecd9a249440f840fe7e [term 1 LEADER]: Becoming Leader. State: Replica: 26870aaaaf054ecd9a249440f840fe7e, State: Running, Role: LEADER
I20260926 01:10:49.860119 11117 consensus_queue.cc:237] T 54febe924d6143b4aa4c3fd7db942ba4 P 26870aaaaf054ecd9a249440f840fe7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26870aaaaf054ecd9a249440f840fe7e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32777 } } peers { permanent_uuid: "a695c7b307c249bd897781eaab8da73f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42293 } } peers { permanent_uuid: "3df9a1da0b21408eab766e5acbe57a31" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42417 } }
I20260926 01:10:49.860926 10637 catalog_manager.cc:5495] T 54febe924d6143b4aa4c3fd7db942ba4 P 26870aaaaf054ecd9a249440f840fe7e reported cstate change: term changed from 0 to 1, leader changed from <none> to 26870aaaaf054ecd9a249440f840fe7e (127.0.98.1). New cstate: current_term: 1 leader_uuid: "26870aaaaf054ecd9a249440f840fe7e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26870aaaaf054ecd9a249440f840fe7e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32777 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a695c7b307c249bd897781eaab8da73f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3df9a1da0b21408eab766e5acbe57a31" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42417 } health_report { overall_health: UNKNOWN } } }
I20260926 01:10:49.885119 11106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:49.882437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58668 (local address 127.0.98.1:32777)
0926 01:10:49.882494 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 01:10:49.882496 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:10:49.882505 (+ 9us) server_negotiation.cc:407] Connection header received
0926 01:10:49.882552 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:10:49.882552 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:10:49.882573 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:10:49.882636 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:10:49.882845 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:49.883356 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:49.884241 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:10:49.884458 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:10:49.884523 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:10:49.884658 (+ 135us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:10:49.884767 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:10:49.884836 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 01:10:49.885008 (+ 172us) server_negotiation.cc:299] Negotiation successful
0926 01:10:49.885032 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 01:10:49.887152 10917 raft_consensus.cc:1261] T 54febe924d6143b4aa4c3fd7db942ba4 P a695c7b307c249bd897781eaab8da73f [term 1 FOLLOWER]: Refusing update from remote peer 26870aaaaf054ecd9a249440f840fe7e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:10:49.887243 11051 raft_consensus.cc:1261] T 54febe924d6143b4aa4c3fd7db942ba4 P 3df9a1da0b21408eab766e5acbe57a31 [term 1 FOLLOWER]: Refusing update from remote peer 26870aaaaf054ecd9a249440f840fe7e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:10:49.887549 11117 consensus_queue.cc:1035] T 54febe924d6143b4aa4c3fd7db942ba4 P 26870aaaaf054ecd9a249440f840fe7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a695c7b307c249bd897781eaab8da73f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:10:49.887704 11122 consensus_queue.cc:1035] T 54febe924d6143b4aa4c3fd7db942ba4 P 26870aaaaf054ecd9a249440f840fe7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3df9a1da0b21408eab766e5acbe57a31" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:10:49.890722 11125 mvcc.cc:205] Tried to move back new op lower bound from 7333417164334530560 to 7333417164227592192. Current Snapshot: MvccSnapshot[applied={T|T < 7333417164334530560}]
I20260926 01:10:49.891110 11126 mvcc.cc:205] Tried to move back new op lower bound from 7333417164334530560 to 7333417164227592192. Current Snapshot: MvccSnapshot[applied={T|T < 7333417164334530560 or (T in {7333417164334530560})}]
I20260926 01:10:49.891243 11127 mvcc.cc:205] Tried to move back new op lower bound from 7333417164334530560 to 7333417164227592192. Current Snapshot: MvccSnapshot[applied={T|T < 7333417164334530560}]
I20260926 01:10:49.893275 10638 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36276:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:10:49.893368 10638 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36276:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:10:49.894250 10638 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fdf82d80753b409c8ff4dd2e8dcef759: Sending DeleteTablet for 3 replicas of tablet 54febe924d6143b4aa4c3fd7db942ba4
I20260926 01:10:49.894784 11031 tablet_service.cc:1514] Processing DeleteTablet for tablet 54febe924d6143b4aa4c3fd7db942ba4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:10:49 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33858
I20260926 01:10:49.894830 10763 tablet_service.cc:1514] Processing DeleteTablet for tablet 54febe924d6143b4aa4c3fd7db942ba4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:10:49 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58652
I20260926 01:10:49.894877 10897 tablet_service.cc:1514] Processing DeleteTablet for tablet 54febe924d6143b4aa4c3fd7db942ba4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:10:49 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44478
I20260926 01:10:49.895066 11135 tablet_replica.cc:330] T 54febe924d6143b4aa4c3fd7db942ba4 P 3df9a1da0b21408eab766e5acbe57a31: stopping tablet replica
I20260926 01:10:49.895138 11137 tablet_replica.cc:330] T 54febe924d6143b4aa4c3fd7db942ba4 P a695c7b307c249bd897781eaab8da73f: stopping tablet replica
I20260926 01:10:49.895211 11135 raft_consensus.cc:2229] T 54febe924d6143b4aa4c3fd7db942ba4 P 3df9a1da0b21408eab766e5acbe57a31 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:10:49.895249 11137 raft_consensus.cc:2229] T 54febe924d6143b4aa4c3fd7db942ba4 P a695c7b307c249bd897781eaab8da73f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:10:49.895344 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 10700
I20260926 01:10:49.895376 11137 raft_consensus.cc:2258] T 54febe924d6143b4aa4c3fd7db942ba4 P a695c7b307c249bd897781eaab8da73f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:10:49.896032 11135 raft_consensus.cc:2258] T 54febe924d6143b4aa4c3fd7db942ba4 P 3df9a1da0b21408eab766e5acbe57a31 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:10:49.896063 11137 ts_tablet_manager.cc:1905] T 54febe924d6143b4aa4c3fd7db942ba4 P a695c7b307c249bd897781eaab8da73f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:10:49.896481 11135 ts_tablet_manager.cc:1905] T 54febe924d6143b4aa4c3fd7db942ba4 P 3df9a1da0b21408eab766e5acbe57a31: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:10:49.898068 11135 ts_tablet_manager.cc:1918] T 54febe924d6143b4aa4c3fd7db942ba4 P 3df9a1da0b21408eab766e5acbe57a31: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:10:49.898109 11137 ts_tablet_manager.cc:1918] T 54febe924d6143b4aa4c3fd7db942ba4 P a695c7b307c249bd897781eaab8da73f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:10:49.898168 11137 log.cc:1196] T 54febe924d6143b4aa4c3fd7db942ba4 P a695c7b307c249bd897781eaab8da73f: Deleting WAL directory at /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384993574762-392-0/minicluster-data/ts-1/wal/wals/54febe924d6143b4aa4c3fd7db942ba4
I20260926 01:10:49.898176 11135 log.cc:1196] T 54febe924d6143b4aa4c3fd7db942ba4 P 3df9a1da0b21408eab766e5acbe57a31: Deleting WAL directory at /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384993574762-392-0/minicluster-data/ts-2/wal/wals/54febe924d6143b4aa4c3fd7db942ba4
I20260926 01:10:49.898435 11135 ts_tablet_manager.cc:1939] T 54febe924d6143b4aa4c3fd7db942ba4 P 3df9a1da0b21408eab766e5acbe57a31: Deleting consensus metadata
I20260926 01:10:49.898435 11137 ts_tablet_manager.cc:1939] T 54febe924d6143b4aa4c3fd7db942ba4 P a695c7b307c249bd897781eaab8da73f: Deleting consensus metadata
I20260926 01:10:49.898878 10624 catalog_manager.cc:4841] TS 3df9a1da0b21408eab766e5acbe57a31 (127.0.98.3:42417): tablet 54febe924d6143b4aa4c3fd7db942ba4 (table test-table [id=b0719f991ae04387bcfa3038ea6dad83]) successfully deleted
I20260926 01:10:49.898973 10624 catalog_manager.cc:4841] TS a695c7b307c249bd897781eaab8da73f (127.0.98.2:42293): tablet 54febe924d6143b4aa4c3fd7db942ba4 (table test-table [id=b0719f991ae04387bcfa3038ea6dad83]) successfully deleted
W20260926 01:10:49.901377 10624 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:32777 (error 108)
I20260926 01:10:49.901912 11108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:10:49.901751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32777 (local address 127.0.0.1:58674)
0926 01:10:49.901808 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0926 01:10:49.901874 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:32777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260926 01:10:49.901938 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 10834
W20260926 01:10:49.902029 10624 catalog_manager.cc:4584] TS 26870aaaaf054ecd9a249440f840fe7e (127.0.98.1:32777): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 54febe924d6143b4aa4c3fd7db942ba4: Network error: Client connection negotiation failed: client connection to 127.0.98.1:32777: connect: Connection refused (error 111)
I20260926 01:10:49.906819 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 10968
I20260926 01:10:49.911654 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 10606
2026-09-26T01:10:49Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2088 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384993574762-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 01:10:49 dist-test-slave-7b22 krb5kdc[11141](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 01:10:49 dist-test-slave-7b22 krb5kdc[11141](info): set up 2 sockets
Sep 26 01:10:49 dist-test-slave-7b22 krb5kdc[11141](info): commencing operation
krb5kdc: starting...
W20260926 01:10:50.879065 392 mini_kdc.cc:121] Time spent starting KDC: real 0.948s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384993574762-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384993574762-392-0/krb5kdc/test-user.keytab.
Sep 26 01:10:50 dist-test-slave-7b22 krb5kdc[11141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385050, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T01:10:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:10:50Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384993574762-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384993574762-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384993574762-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384993574762-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384993574762-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384993574762-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260926 01:10:51.058203 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384993574762-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskzzHEKz/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384993574762-392-0/minicluster-data/postgres -l logfile start
2026-09-26 01:10:53.697 UTC [11171] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-26 01:10:53.697 UTC [11171] LOG: listening on IPv4 address "127.0.98.20", port 54433
2026-09-26 01:10:53.701 UTC [11171] LOG: listening on Unix socket "/tmp/.s.PGSQL.54433"
2026-09-26 01:10:53.729 UTC [11174] LOG: database system was shut down at 2026-09-26 01:10:52 UTC
2026-09-26 01:10:53.733 UTC [11171] LOG: database system is ready to accept connections
I20260926 01:10:54.614367 392 mini_postgres.cc:96] Postgres bound to 54433
2026-09-26 01:10:54.620 UTC [11182] FATAL: database "slave" does not exist
127.0.98.20:54433 - accepting connections
I20260926 01:10:54.620535 392 mini_ranger.cc:161] Starting Ranger...
I20260926 01:10:54.635167 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260926 01:10:55.180037 392 mini_ranger.cc:87] Created ranger Postgres database
I20260926 01:10:55.180127 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384993574762-392-0/minicluster-data/ranger-admin
2026-09-26 01:10:55,439 [I] DB FLAVOR :POSTGRES
2026-09-26 01:10:55,439 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 01:10:55,439 [I] Checking connection..
2026-09-26 01:10:55,439 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzzHEKz/build/release/bin/postgresql.jar:/tmp/dist-test-taskzzHEKz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54433/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-26 01:10:55,673 [I] Checking connection passed.
2026-09-26 01:10:55,674 [I] --------- Verifying version history table ---------
2026-09-26 01:10:55,674 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzzHEKz/build/release/bin/postgresql.jar:/tmp/dist-test-taskzzHEKz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54433/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 01:10:55,937 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 01:10:55,938 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzzHEKz/build/release/bin/postgresql.jar:/tmp/dist-test-taskzzHEKz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54433/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 01:10:56,197 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 01:10:56,197 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-26 01:10:56,197 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzzHEKz/build/release/bin/postgresql.jar:/tmp/dist-test-taskzzHEKz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54433/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskzzHEKz/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-26 01:10:56.429 UTC [11246] WARNING: there is no transaction in progress
2026-09-26 01:10:56,434 [I] create_dbversion_catalog.sql file imported successfully
2026-09-26 01:10:56,435 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzzHEKz/build/release/bin/postgresql.jar:/tmp/dist-test-taskzzHEKz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54433/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 01:10:56,704 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-26 01:10:56,704 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 01:10:56,704 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzzHEKz/build/release/bin/postgresql.jar:/tmp/dist-test-taskzzHEKz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54433/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-26 01:10:56,940 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzzHEKz/build/release/bin/postgresql.jar:/tmp/dist-test-taskzzHEKz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54433/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-26 01:10:57,198 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzzHEKz/build/release/bin/postgresql.jar:/tmp/dist-test-taskzzHEKz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54433/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-7b22.c.gcp-upstream.internal','N') ;"
2026-09-26 01:10:57,427 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzzHEKz/build/release/bin/postgresql.jar:/tmp/dist-test-taskzzHEKz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54433/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 01:10:57,687 [I] Table x_portal_user does not exist in database ranger
2026-09-26 01:10:57,687 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzzHEKz/build/release/bin/postgresql.jar:/tmp/dist-test-taskzzHEKz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54433/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 01:10:57,946 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-26 01:10:57,946 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzzHEKz/build/release/bin/postgresql.jar:/tmp/dist-test-taskzzHEKz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54433/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 01:10:58,197 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzzHEKz/build/release/bin/postgresql.jar:/tmp/dist-test-taskzzHEKz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54433/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 01:10:58,426 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-26 01:10:58,426 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzzHEKz/build/release/bin/postgresql.jar:/tmp/dist-test-taskzzHEKz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54433/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskzzHEKz/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-26 01:10:59.436 UTC [11354] WARNING: there is no transaction in progress
2026-09-26 01:10:59.452 UTC [11354] WARNING: there is no transaction in progress
2026-09-26 01:10:59.469 UTC [11354] WARNING: there is no transaction in progress
2026-09-26 01:10:59.485 UTC [11354] WARNING: there is no transaction in progress
2026-09-26 01:10:59.502 UTC [11354] WARNING: there is no transaction in progress
2026-09-26 01:10:59.521 UTC [11354] WARNING: there is no transaction in progress
2026-09-26 01:10:59.590 UTC [11354] WARNING: there is no transaction in progress
2026-09-26 01:10:59.597 UTC [11354] WARNING: there is no transaction in progress
2026-09-26 01:10:59.613 UTC [11354] WARNING: there is no transaction in progress
2026-09-26 01:10:59.624 UTC [11354] WARNING: there is no transaction in progress
2026-09-26 01:10:59.635 UTC [11354] WARNING: there is no transaction in progress
2026-09-26 01:10:59.650 UTC [11354] WARNING: there is no transaction in progress
2026-09-26 01:10:59.660 UTC [11354] WARNING: there is no transaction in progress
2026-09-26 01:10:59.666 UTC [11354] WARNING: there is no transaction in progress
2026-09-26 01:10:59.673 UTC [11354] WARNING: there is no transaction in progress
2026-09-26 01:10:59.681 UTC [11354] WARNING: there is no transaction in progress
2026-09-26 01:11:00,398 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-26 01:11:00,398 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzzHEKz/build/release/bin/postgresql.jar:/tmp/dist-test-taskzzHEKz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54433/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-26 01:11:00,632 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-26 01:11:00,632 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzzHEKz/build/release/bin/postgresql.jar:/tmp/dist-test-taskzzHEKz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54433/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 01:11:00,882 [I] Table x_portal_user already exists in database 'ranger'
2026-09-26 01:11:00,882 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzzHEKz/build/release/bin/postgresql.jar:/tmp/dist-test-taskzzHEKz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54433/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 01:11:01,118 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-26 01:11:01,118 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzzHEKz/build/release/bin/postgresql.jar:/tmp/dist-test-taskzzHEKz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54433/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 01:11:01,377 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzzHEKz/build/release/bin/postgresql.jar:/tmp/dist-test-taskzzHEKz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54433/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 01:11:01,621 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzzHEKz/build/release/bin/postgresql.jar:/tmp/dist-test-taskzzHEKz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54433/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-7b22.c.gcp-upstream.internal';"
2026-09-26 01:11:01,860 [I] CORE_DB_SCHEMA import status has been updated
2026-09-26 01:11:01,860 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzzHEKz/build/release/bin/postgresql.jar:/tmp/dist-test-taskzzHEKz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54433/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-26 01:11:02,094 [I] DB_PATCHES have already been applied
I20260926 01:11:02.100003 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384993574762-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskzzHEKz/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskzzHEKz/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskzzHEKz/build/release/bin/postgresql.jar:/tmp/dist-test-taskzzHEKz/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260926 01:11:02.100073 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260926 01:11:02.103047 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:60531
Sep 26, 2026 1:11:02 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 1:11:02 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 1:11:02 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384993574762-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 26, 2026 1:11:02 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384993574762-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 26, 2026 1:11:02 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384993574762-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 26, 2026 1:11:02 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384993574762-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 26, 2026 1:11:02 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260926 01:11:03.113490 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.493s user 0.001s sys 0.007s
Sep 26, 2026 1:11:23 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 26, 2026 1:11:23 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 26, 2026 1:11:23 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 26, 2026 1:11:23 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 26, 2026 1:11:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 26, 2026 1:11:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 26, 2026 1:11:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 26, 2026 1:11:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 26, 2026 1:11:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 26, 2026 1:11:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 26, 2026 1:11:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 26, 2026 1:11:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 26, 2026 1:11:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 26, 2026 1:11:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 26, 2026 1:11:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 26, 2026 1:11:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 26, 2026 1:11:23 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 26, 2026 1:11:23 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260926 01:11:24.589807 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384993574762-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384993574762-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384993574762-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384993574762-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 26 01:11:24 dist-test-slave-7b22 krb5kdc[11141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385084, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260926 01:11:24.673398 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260926 01:11:24.682408 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260926 01:11:25.664064 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260926 01:11:25.664147 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384993574762-392-0/minicluster-data/ranger-kms
I20260926 01:11:25.664165 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:54433
2026-09-26 01:11:25,832 [I] DB FLAVOR :POSTGRES
2026-09-26 01:11:25,832 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 01:11:25,833 [I] Checking connection
2026-09-26 01:11:25,833 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzzHEKz/build/release/bin/postgresql.jar:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384993574762-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54433/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-26 01:11:26,098 [I] connection success
2026-09-26 01:11:26,098 [I] --------- Verifying Ranger DB tables ---------
2026-09-26 01:11:26,098 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-26 01:11:26,098 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzzHEKz/build/release/bin/postgresql.jar:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384993574762-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54433/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-26 01:11:26,353 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-26 01:11:26,354 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 01:11:26,354 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-26 01:11:26,354 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzzHEKz/build/release/bin/postgresql.jar:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384993574762-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54433/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384993574762-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-26 01:11:26,620 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260926 01:11:27.119339 392 mini_ranger_kms.cc:310] Created kms service
I20260926 01:11:27.344643 392 mini_ranger_kms.cc:326] Created kudu user
I20260926 01:11:27.394309 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 26 01:11:27 dist-test-slave-7b22 krb5kdc[11141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385084, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 01:11:27.845028 392 mini_ranger_kms.cc:384] Added ranger policy
I20260926 01:11:27.845309 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384993574762-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskzzHEKz/build/release/bin/postgresql.jar:/tmp/dist-test-taskzzHEKz/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskzzHEKz/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260926 01:11:27.845340 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260926 01:11:27.848831 392 mini_ranger_kms.cc:276] Ranger KMS PID: 11534
I20260926 01:11:27.848948 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:51307
Sep 26, 2026 1:11:28 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 1:11:28 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 1:11:28 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384993574762-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 26, 2026 1:11:28 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384993574762-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 26, 2026 1:11:29 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384993574762-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 26, 2026 1:11:29 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260926 01:11:29.836349 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.163s user 0.003s sys 0.147s
I20260926 01:11:29.836434 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260926 01:11:29.836515 392 mini_ranger_kms.cc:401] 127.0.98.20:51307/kms/v1/keys
Sep 26 01:11:30 dist-test-slave-7b22 krb5kdc[11141](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790385090, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 26 01:11:30 dist-test-slave-7b22 krb5kdc[11141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385090, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 26 01:11:30 dist-test-slave-7b22 krb5kdc[11141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385090, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 26 01:11:33 dist-test-slave-7b22 krb5kdc[11141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385093, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384993574762-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384993574762-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 01:11:33.288851 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384993574762-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384993574762-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384993574762-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43719
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384993574762-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43719
--ranger_config_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384993574762-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:51307/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:11:33.392680 11585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:11:33.392901 11585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:11:33.392939 11585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:11:33.394168 11585 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:11:33.394209 11585 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:11:33.394227 11585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:11:33.394243 11585 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:11:33.394259 11585 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:11:33.395833 11585 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384993574762-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384993574762-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:51307/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43719
--ranger_config_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384993574762-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384993574762-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43719
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11585
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384993574762-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:11:33.396080 11585 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:11:33.396282 11585 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 01:11:33.401222 11585 server_base.cc:1034] running on GCE node
W20260926 01:11:33.401293 11593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:11:33.401405 11590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:11:33.401311 11591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:11:33.401789 11585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:11:33.402101 11585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:11:33.403239 11585 hybrid_clock.cc:648] HybridClock initialized: now 1790385093403219 us; error 37 us; skew 500 ppm
Sep 26 01:11:33 dist-test-slave-7b22 krb5kdc[11141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385093, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:11:33.409068 11585 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 01:11:33.409544 11585 webserver.cc:460] Webserver started at http://127.0.98.62:46087/ using document root <none> and password file <none>
I20260926 01:11:33.409740 11585 fs_manager.cc:359] Metadata directory not provided
I20260926 01:11:33.409780 11585 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:11:33.409940 11585 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 01:11:33 dist-test-slave-7b22 krb5kdc[11141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385093, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 01:11:33.464478 11585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384993574762-392-0/minicluster-data/master-0/data/instance:
uuid: "3541b67264cf4dafb3ea63b68bb28c08"
format_stamp: "Formatted at 2026-09-26 01:11:33 on dist-test-slave-7b22"
server_key: "09dfa175befd099be8a31541eb15b2be"
server_key_iv: "7027fe40a3ea3e77c9f7dd3206828845"
server_key_version: "kuduclusterkey@0"
I20260926 01:11:33.464953 11585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384993574762-392-0/minicluster-data/master-0/wal/instance:
uuid: "3541b67264cf4dafb3ea63b68bb28c08"
format_stamp: "Formatted at 2026-09-26 01:11:33 on dist-test-slave-7b22"
server_key: "09dfa175befd099be8a31541eb15b2be"
server_key_iv: "7027fe40a3ea3e77c9f7dd3206828845"
server_key_version: "kuduclusterkey@0"
I20260926 01:11:33.466749 11585 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260926 01:11:33.478729 11600 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:11:33.478940 11585 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 01:11:33.479034 11585 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384993574762-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384993574762-392-0/minicluster-data/master-0/wal
uuid: "3541b67264cf4dafb3ea63b68bb28c08"
format_stamp: "Formatted at 2026-09-26 01:11:33 on dist-test-slave-7b22"
server_key: "09dfa175befd099be8a31541eb15b2be"
server_key_iv: "7027fe40a3ea3e77c9f7dd3206828845"
server_key_version: "kuduclusterkey@0"
I20260926 01:11:33.479138 11585 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384993574762-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384993574762-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384993574762-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:11:33.488451 11585 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:11:33.490216 11585 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:11:33.490350 11585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:11:33.495254 11585 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43719
I20260926 01:11:33.495293 11652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43719 every 8 connection(s)
I20260926 01:11:33.495632 11585 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
I20260926 01:11:33.496865 11653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:11:33.499995 11653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3541b67264cf4dafb3ea63b68bb28c08: Bootstrap starting.
I20260926 01:11:33.500699 11653 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3541b67264cf4dafb3ea63b68bb28c08: Neither blocks nor log segments found. Creating new log.
I20260926 01:11:33.500984 11653 log.cc:824] T 00000000000000000000000000000000 P 3541b67264cf4dafb3ea63b68bb28c08: Log is configured to *not* fsync() on all Append() calls
I20260926 01:11:33.501775 11653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3541b67264cf4dafb3ea63b68bb28c08: No bootstrap required, opened a new log
I20260926 01:11:33.504608 11653 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3541b67264cf4dafb3ea63b68bb28c08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3541b67264cf4dafb3ea63b68bb28c08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43719 } }
I20260926 01:11:33.504805 11653 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3541b67264cf4dafb3ea63b68bb28c08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:11:33.504837 11653 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3541b67264cf4dafb3ea63b68bb28c08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3541b67264cf4dafb3ea63b68bb28c08, State: Initialized, Role: FOLLOWER
I20260926 01:11:33.504982 11653 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3541b67264cf4dafb3ea63b68bb28c08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3541b67264cf4dafb3ea63b68bb28c08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43719 } }
I20260926 01:11:33.505084 11653 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3541b67264cf4dafb3ea63b68bb28c08 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:11:33.505129 11653 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3541b67264cf4dafb3ea63b68bb28c08 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:11:33.505213 11653 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3541b67264cf4dafb3ea63b68bb28c08 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:11:33.505215 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 11585
I20260926 01:11:33.505308 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384993574762-392-0/minicluster-data/master-0/wal/instance
I20260926 01:11:33.505523 392 external_mini_cluster.cc:1428] Setting key 23f58b5f94d723b1c2893f6bc13f9894
I20260926 01:11:33.505977 11653 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3541b67264cf4dafb3ea63b68bb28c08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3541b67264cf4dafb3ea63b68bb28c08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43719 } }
I20260926 01:11:33.506110 11653 leader_election.cc:302] T 00000000000000000000000000000000 P 3541b67264cf4dafb3ea63b68bb28c08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3541b67264cf4dafb3ea63b68bb28c08; no voters:
I20260926 01:11:33.506305 11653 leader_election.cc:288] T 00000000000000000000000000000000 P 3541b67264cf4dafb3ea63b68bb28c08 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:11:33.506331 11658 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3541b67264cf4dafb3ea63b68bb28c08 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:11:33.506575 11658 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3541b67264cf4dafb3ea63b68bb28c08 [term 1 LEADER]: Becoming Leader. State: Replica: 3541b67264cf4dafb3ea63b68bb28c08, State: Running, Role: LEADER
I20260926 01:11:33.506683 11653 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3541b67264cf4dafb3ea63b68bb28c08 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:11:33.506708 11658 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3541b67264cf4dafb3ea63b68bb28c08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3541b67264cf4dafb3ea63b68bb28c08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43719 } }
I20260926 01:11:33.509203 11659 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3541b67264cf4dafb3ea63b68bb28c08 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3541b67264cf4dafb3ea63b68bb28c08" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3541b67264cf4dafb3ea63b68bb28c08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43719 } } }
I20260926 01:11:33.509290 11659 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3541b67264cf4dafb3ea63b68bb28c08 [sys.catalog]: This master's current role is: LEADER
I20260926 01:11:33.509428 11659 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3541b67264cf4dafb3ea63b68bb28c08 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3541b67264cf4dafb3ea63b68bb28c08. Latest consensus state: current_term: 1 leader_uuid: "3541b67264cf4dafb3ea63b68bb28c08" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3541b67264cf4dafb3ea63b68bb28c08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43719 } } }
I20260926 01:11:33.509480 11659 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3541b67264cf4dafb3ea63b68bb28c08 [sys.catalog]: This master's current role is: LEADER
I20260926 01:11:33.509729 11672 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:11:33.510033 11653 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384993574762-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 01:11:33.510356 11672 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 26 01:11:33 dist-test-slave-7b22 krb5kdc[11141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385093, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:11:33.517196 11657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:11:33.506141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48272 (local address 127.0.98.62:43719)
0926 01:11:33.506361 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0926 01:11:33.506371 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 01:11:33.506404 (+ 33us) server_negotiation.cc:407] Connection header received
0926 01:11:33.506528 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:11:33.506536 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:11:33.506629 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:11:33.506771 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:11:33.507504 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:11:33.508518 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:11:33.509580 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:11:33.510223 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:11:33.510323 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:11:33.513398 (+ 3075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:11:33.513435 (+ 37us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:11:33.513435 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:11:33.513475 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:11:33.515172 (+ 1697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:11:33.515563 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:11:33.515567 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:11:33.515567 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:11:33.515660 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:11:33.515843 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:11:33.515844 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:11:33.515845 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:11:33.516176 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:11:33.516305 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0926 01:11:33.516801 (+ 496us) server_negotiation.cc:299] Negotiation successful
0926 01:11:33.516969 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":77,"threads_started":1}
I20260926 01:11:33.521332 11672 catalog_manager.cc:1347] Generated new cluster ID: c2f0495b7bfb4a6395f858483026b583
I20260926 01:11:33.521392 11672 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 01:11:33.553050 11672 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:11:33.553890 11672 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:11:33.558704 11672 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3541b67264cf4dafb3ea63b68bb28c08: Generated new TSK 0
I20260926 01:11:33.558976 11672 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384993574762-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384993574762-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 01:11:34.284929 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384993574762-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384993574762-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384993574762-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384993574762-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:51307/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43719
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384993574762-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:11:34.387943 11698 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:11:34.388163 11698 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:11:34.388196 11698 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:11:34.389430 11698 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:11:34.389475 11698 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:11:34.389513 11698 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 01:11:34.391589 11698 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384993574762-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384993574762-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:51307/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384993574762-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384993574762-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43719
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11698
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384993574762-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:11:34.391834 11698 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:11:34.392218 11698 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:11:34.397324 11703 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:11:34.397331 11704 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:11:34.397415 11706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:11:34.397541 11698 server_base.cc:1034] running on GCE node
I20260926 01:11:34.397776 11698 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:11:34.398047 11698 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:11:34.399209 11698 hybrid_clock.cc:648] HybridClock initialized: now 1790385094399188 us; error 37 us; skew 500 ppm
Sep 26 01:11:34 dist-test-slave-7b22 krb5kdc[11141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385094, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:11:34.402114 11698 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 01:11:34.402628 11698 webserver.cc:460] Webserver started at http://127.0.98.1:43217/ using document root <none> and password file <none>
I20260926 01:11:34.402860 11698 fs_manager.cc:359] Metadata directory not provided
I20260926 01:11:34.402915 11698 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:11:34.403053 11698 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 01:11:34 dist-test-slave-7b22 krb5kdc[11141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385094, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 01:11:34.419085 11698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384993574762-392-0/minicluster-data/ts-0/data/instance:
uuid: "0624dd89dbdf48daaf5df4c0bc5f29da"
format_stamp: "Formatted at 2026-09-26 01:11:34 on dist-test-slave-7b22"
server_key: "971d30e1e9621ae5fef38eba5a6d037f"
server_key_iv: "ba3860a543ceb9c87f0684c89370ef3c"
server_key_version: "kuduclusterkey@0"
I20260926 01:11:34.419543 11698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384993574762-392-0/minicluster-data/ts-0/wal/instance:
uuid: "0624dd89dbdf48daaf5df4c0bc5f29da"
format_stamp: "Formatted at 2026-09-26 01:11:34 on dist-test-slave-7b22"
server_key: "971d30e1e9621ae5fef38eba5a6d037f"
server_key_iv: "ba3860a543ceb9c87f0684c89370ef3c"
server_key_version: "kuduclusterkey@0"
I20260926 01:11:34.423290 11698 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.001s
I20260926 01:11:34.435868 11713 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:11:34.436056 11698 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 01:11:34.436144 11698 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384993574762-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384993574762-392-0/minicluster-data/ts-0/wal
uuid: "0624dd89dbdf48daaf5df4c0bc5f29da"
format_stamp: "Formatted at 2026-09-26 01:11:34 on dist-test-slave-7b22"
server_key: "971d30e1e9621ae5fef38eba5a6d037f"
server_key_iv: "ba3860a543ceb9c87f0684c89370ef3c"
server_key_version: "kuduclusterkey@0"
I20260926 01:11:34.436236 11698 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384993574762-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384993574762-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384993574762-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:11:34.463536 11698 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:11:34.465397 11698 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:11:34.465550 11698 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:11:34.465775 11698 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:11:34.466145 11698 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:11:34.466217 11698 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:11:34.466272 11698 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:11:34.466305 11698 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:11:34.473626 11698 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38867
I20260926 01:11:34.473718 11828 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38867 every 8 connection(s)
I20260926 01:11:34.474053 11698 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384993574762-392-0/minicluster-data/ts-0/data/info.pb
I20260926 01:11:34.481043 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 11698
I20260926 01:11:34.481153 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384993574762-392-0/minicluster-data/ts-0/wal/instance
I20260926 01:11:34.481344 392 external_mini_cluster.cc:1428] Setting key bd371acbc34830cfd4d9a49070472955
Sep 26 01:11:34 dist-test-slave-7b22 krb5kdc[11141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385094, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:11:34.486333 11831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:11:34.475228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43719 (local address 127.0.98.1:34613)
0926 01:11:34.475417 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0926 01:11:34.475469 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 01:11:34.476096 (+ 627us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:11:34.476525 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:11:34.476530 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:11:34.476770 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:11:34.477048 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:11:34.477054 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:11:34.478091 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:11:34.478091 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:11:34.478908 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:11:34.478910 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:11:34.479382 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:11:34.479384 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:11:34.479540 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:11:34.480078 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:11:34.480086 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:11:34.481730 (+ 1644us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:11:34.484818 (+ 3088us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:11:34.484826 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:11:34.484828 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:11:34.485074 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:11:34.485261 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:11:34.485261 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:11:34.485262 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:11:34.485379 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:11:34.485620 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:11:34.485627 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:11:34.485785 (+ 158us) client_negotiation.cc:769] Sending connection context
0926 01:11:34.485925 (+ 140us) client_negotiation.cc:240] Negotiation successful
0926 01:11:34.486100 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":85,"threads_started":1}
I20260926 01:11:34.487113 11832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:11:34.475247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34613 (local address 127.0.98.62:43719)
0926 01:11:34.476277 (+ 1030us) server_negotiation.cc:206] Beginning negotiation
0926 01:11:34.476280 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:11:34.476287 (+ 7us) server_negotiation.cc:407] Connection header received
0926 01:11:34.476311 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:11:34.476312 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:11:34.476325 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:11:34.476390 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:11:34.477166 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:11:34.477989 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:11:34.478995 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:11:34.479294 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:11:34.479377 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:11:34.481849 (+ 2472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:11:34.481851 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:11:34.481851 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:11:34.481860 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:11:34.484738 (+ 2878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:11:34.485143 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:11:34.485144 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:11:34.485145 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:11:34.485207 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:11:34.485434 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:11:34.485435 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:11:34.485435 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:11:34.485564 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:11:34.485620 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 01:11:34.486908 (+ 1288us) server_negotiation.cc:299] Negotiation successful
0926 01:11:34.487017 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1002,"thread_start_us":78,"threads_started":1}
I20260926 01:11:34.487697 11829 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43719
I20260926 01:11:34.487794 11829 heartbeater.cc:464] Registering TS with master...
I20260926 01:11:34.487998 11829 heartbeater.cc:511] Master 127.0.98.62:43719 requested a full tablet report, sending...
I20260926 01:11:34.488958 11617 ts_manager.cc:194] Registered new tserver with Master: 0624dd89dbdf48daaf5df4c0bc5f29da (127.0.98.1:38867)
I20260926 01:11:34.490060 11617 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:34613
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384993574762-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384993574762-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 01:11:34.517079 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384993574762-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384993574762-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384993574762-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384993574762-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:51307/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43719
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384993574762-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:11:34.617272 11835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:11:34.617486 11835 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:11:34.617519 11835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:11:34.618790 11835 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:11:34.618850 11835 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:11:34.618888 11835 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 01:11:34.620566 11835 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384993574762-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384993574762-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:51307/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384993574762-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384993574762-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43719
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11835
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384993574762-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:11:34.620848 11835 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:11:34.621086 11835 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:11:34.626230 11845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:11:34.626359 11842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:11:34.626578 11843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:11:34.627110 11835 server_base.cc:1034] running on GCE node
I20260926 01:11:34.627353 11835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:11:34.627663 11835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:11:34.630955 11835 hybrid_clock.cc:648] HybridClock initialized: now 1790385094630918 us; error 49 us; skew 500 ppm
Sep 26 01:11:34 dist-test-slave-7b22 krb5kdc[11141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385094, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:11:34.634632 11835 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 01:11:34.635221 11835 webserver.cc:460] Webserver started at http://127.0.98.2:41989/ using document root <none> and password file <none>
I20260926 01:11:34.635517 11835 fs_manager.cc:359] Metadata directory not provided
I20260926 01:11:34.635583 11835 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:11:34.635771 11835 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 01:11:34 dist-test-slave-7b22 krb5kdc[11141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385094, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 26 01:11:34 dist-test-slave-7b22 krb5kdc[11141](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790385094, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:11:34.648968 11835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384993574762-392-0/minicluster-data/ts-1/data/instance:
uuid: "8e09e749c5dd4e00ad5cea18e0080c41"
format_stamp: "Formatted at 2026-09-26 01:11:34 on dist-test-slave-7b22"
server_key: "3910217460cbd91a1a1ca84e5d37232b"
server_key_iv: "3cf7453d0d9a0053a7c8377cf644087c"
server_key_version: "kuduclusterkey@0"
I20260926 01:11:34.649477 11835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384993574762-392-0/minicluster-data/ts-1/wal/instance:
uuid: "8e09e749c5dd4e00ad5cea18e0080c41"
format_stamp: "Formatted at 2026-09-26 01:11:34 on dist-test-slave-7b22"
server_key: "3910217460cbd91a1a1ca84e5d37232b"
server_key_iv: "3cf7453d0d9a0053a7c8377cf644087c"
server_key_version: "kuduclusterkey@0"
I20260926 01:11:34.651770 11835 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260926 01:11:34.670121 11852 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:11:34.670395 11835 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 01:11:34.670499 11835 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384993574762-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384993574762-392-0/minicluster-data/ts-1/wal
uuid: "8e09e749c5dd4e00ad5cea18e0080c41"
format_stamp: "Formatted at 2026-09-26 01:11:34 on dist-test-slave-7b22"
server_key: "3910217460cbd91a1a1ca84e5d37232b"
server_key_iv: "3cf7453d0d9a0053a7c8377cf644087c"
server_key_version: "kuduclusterkey@0"
I20260926 01:11:34.670605 11835 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384993574762-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384993574762-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384993574762-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:11:34.692790 11835 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:11:34.695133 11835 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:11:34.695325 11835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:11:34.695604 11835 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:11:34.696036 11835 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:11:34.696128 11835 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:11:34.696203 11835 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:11:34.696259 11835 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:11:34.704003 11835 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44623
I20260926 01:11:34.704082 11965 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44623 every 8 connection(s)
I20260926 01:11:34.704520 11835 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384993574762-392-0/minicluster-data/ts-1/data/info.pb
I20260926 01:11:34.708964 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 11835
I20260926 01:11:34.709080 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384993574762-392-0/minicluster-data/ts-1/wal/instance
I20260926 01:11:34.709249 392 external_mini_cluster.cc:1428] Setting key 133a0b5e4ae1f33030368264771d0901
Sep 26 01:11:34 dist-test-slave-7b22 krb5kdc[11141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385094, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:11:34.718730 11832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:11:34.706887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44639 (local address 127.0.98.62:43719)
0926 01:11:34.706957 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0926 01:11:34.706959 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:11:34.707175 (+ 216us) server_negotiation.cc:407] Connection header received
0926 01:11:34.707893 (+ 718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:11:34.707895 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:11:34.707908 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:11:34.707969 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:11:34.708698 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:11:34.709491 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:11:34.711643 (+ 2152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:11:34.711894 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:11:34.711962 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:11:34.715382 (+ 3420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:11:34.715384 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:11:34.715384 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:11:34.715393 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:11:34.717057 (+ 1664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:11:34.717576 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:11:34.717578 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:11:34.717578 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:11:34.717638 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:11:34.717933 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:11:34.717934 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:11:34.717934 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:11:34.718083 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:11:34.718125 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0926 01:11:34.718524 (+ 399us) server_negotiation.cc:299] Negotiation successful
0926 01:11:34.718626 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260926 01:11:34.719547 11968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:11:34.706848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43719 (local address 127.0.98.2:44639)
0926 01:11:34.707023 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 01:11:34.707075 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 01:11:34.707743 (+ 668us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:11:34.708045 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:11:34.708050 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:11:34.708305 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:11:34.708587 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:11:34.708593 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:11:34.709590 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:11:34.709591 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:11:34.711554 (+ 1963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:11:34.711557 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:11:34.712028 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:11:34.712029 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:11:34.712146 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:11:34.713582 (+ 1436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:11:34.713593 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:11:34.715260 (+ 1667us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:11:34.717161 (+ 1901us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:11:34.717168 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:11:34.717171 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:11:34.717487 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:11:34.717705 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:11:34.717705 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:11:34.717706 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:11:34.717860 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:11:34.718169 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:11:34.718175 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:11:34.718361 (+ 186us) client_negotiation.cc:769] Sending connection context
0926 01:11:34.719094 (+ 733us) client_negotiation.cc:240] Negotiation successful
0926 01:11:34.719279 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":91,"threads_started":1}
I20260926 01:11:34.719996 11966 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43719
I20260926 01:11:34.720150 11966 heartbeater.cc:464] Registering TS with master...
I20260926 01:11:34.720391 11966 heartbeater.cc:511] Master 127.0.98.62:43719 requested a full tablet report, sending...
I20260926 01:11:34.720796 11617 ts_manager.cc:194] Registered new tserver with Master: 8e09e749c5dd4e00ad5cea18e0080c41 (127.0.98.2:44623)
I20260926 01:11:34.721298 11617 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:44639
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384993574762-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384993574762-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 01:11:34.739141 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384993574762-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384993574762-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384993574762-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384993574762-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:51307/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43719
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384993574762-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:11:34.862794 11972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:11:34.863101 11972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:11:34.863145 11972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:11:34.864924 11972 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:11:34.864985 11972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:11:34.865027 11972 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 01:11:34.867321 11972 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384993574762-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384993574762-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:51307/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384993574762-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384993574762-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43719
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11972
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384993574762-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:11:34.867743 11972 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:11:34.868057 11972 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:11:34.874591 11980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:11:34.875322 11978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:11:34.875442 11977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:11:34.876498 11972 server_base.cc:1034] running on GCE node
I20260926 01:11:34.876819 11972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:11:34.877106 11972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:11:34.878270 11972 hybrid_clock.cc:648] HybridClock initialized: now 1790385094878245 us; error 37 us; skew 500 ppm
Sep 26 01:11:34 dist-test-slave-7b22 krb5kdc[11141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385094, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:11:34.881682 11972 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 01:11:34.882318 11972 webserver.cc:460] Webserver started at http://127.0.98.3:34555/ using document root <none> and password file <none>
I20260926 01:11:34.882658 11972 fs_manager.cc:359] Metadata directory not provided
I20260926 01:11:34.882782 11972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:11:34.882941 11972 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 01:11:34 dist-test-slave-7b22 krb5kdc[11141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385094, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 01:11:34.915887 11972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384993574762-392-0/minicluster-data/ts-2/data/instance:
uuid: "628e7d55677e498ab92a9b1ac0ba055a"
format_stamp: "Formatted at 2026-09-26 01:11:34 on dist-test-slave-7b22"
server_key: "474e37aefe36d94bba9e4053e5913af1"
server_key_iv: "ea591aa3ebeae0a7356e5287a9b4d77e"
server_key_version: "kuduclusterkey@0"
I20260926 01:11:34.916321 11972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384993574762-392-0/minicluster-data/ts-2/wal/instance:
uuid: "628e7d55677e498ab92a9b1ac0ba055a"
format_stamp: "Formatted at 2026-09-26 01:11:34 on dist-test-slave-7b22"
server_key: "474e37aefe36d94bba9e4053e5913af1"
server_key_iv: "ea591aa3ebeae0a7356e5287a9b4d77e"
server_key_version: "kuduclusterkey@0"
I20260926 01:11:34.918118 11972 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.001s
I20260926 01:11:34.938004 11987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:11:34.938285 11972 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.001s sys 0.000s
I20260926 01:11:34.938388 11972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384993574762-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384993574762-392-0/minicluster-data/ts-2/wal
uuid: "628e7d55677e498ab92a9b1ac0ba055a"
format_stamp: "Formatted at 2026-09-26 01:11:34 on dist-test-slave-7b22"
server_key: "474e37aefe36d94bba9e4053e5913af1"
server_key_iv: "ea591aa3ebeae0a7356e5287a9b4d77e"
server_key_version: "kuduclusterkey@0"
I20260926 01:11:34.938509 11972 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384993574762-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384993574762-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384993574762-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:11:34.980186 11972 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:11:34.982765 11972 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:11:34.983026 11972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:11:34.983309 11972 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:11:34.983732 11972 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:11:34.983786 11972 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:11:34.983819 11972 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:11:34.983835 11972 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:11:34.991916 11972 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40757
I20260926 01:11:34.992115 12100 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40757 every 8 connection(s)
I20260926 01:11:34.992411 11972 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384993574762-392-0/minicluster-data/ts-2/data/info.pb
I20260926 01:11:34.996731 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 11972
I20260926 01:11:34.996853 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384993574762-392-0/minicluster-data/ts-2/wal/instance
I20260926 01:11:34.997038 392 external_mini_cluster.cc:1428] Setting key 6d641d84d41cf36190b46a79cfbb10db
Sep 26 01:11:34 dist-test-slave-7b22 krb5kdc[11141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385094, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:11:35.004365 11832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:11:34.993294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56739 (local address 127.0.98.62:43719)
0926 01:11:34.993344 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 01:11:34.993346 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:11:34.993583 (+ 237us) server_negotiation.cc:407] Connection header received
0926 01:11:34.994304 (+ 721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:11:34.994306 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:11:34.994319 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:11:34.994351 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:11:34.995213 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:11:34.995959 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:11:34.997888 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:11:34.998174 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:11:34.998223 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:11:35.000871 (+ 2648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:11:35.000874 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:11:35.000874 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:11:35.000883 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:11:35.002669 (+ 1786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:11:35.003213 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:11:35.003215 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:11:35.003215 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:11:35.003278 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:11:35.003572 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:11:35.003573 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:11:35.003574 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:11:35.003718 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:11:35.003760 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0926 01:11:35.004174 (+ 414us) server_negotiation.cc:299] Negotiation successful
0926 01:11:35.004264 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 01:11:35.004935 12103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:11:34.993260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43719 (local address 127.0.98.3:56739)
0926 01:11:34.993423 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 01:11:34.993473 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 01:11:34.994167 (+ 694us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:11:34.994544 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:11:34.994549 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:11:34.994807 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:11:34.995093 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:11:34.995099 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:11:34.996087 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:11:34.996089 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:11:34.997762 (+ 1673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:11:34.997765 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:11:34.998321 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:11:34.998322 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:11:34.998470 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:11:34.999102 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:11:34.999112 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:11:35.000748 (+ 1636us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:11:35.002772 (+ 2024us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:11:35.002781 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:11:35.002784 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:11:35.003118 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:11:35.003350 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:11:35.003351 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:11:35.003352 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:11:35.003495 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:11:35.003802 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:11:35.003809 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:11:35.003999 (+ 190us) client_negotiation.cc:769] Sending connection context
0926 01:11:35.004520 (+ 521us) client_negotiation.cc:240] Negotiation successful
0926 01:11:35.004696 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":77,"threads_started":1}
I20260926 01:11:35.005412 12101 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43719
I20260926 01:11:35.005529 12101 heartbeater.cc:464] Registering TS with master...
I20260926 01:11:35.005731 12101 heartbeater.cc:511] Master 127.0.98.62:43719 requested a full tablet report, sending...
I20260926 01:11:35.006198 11617 ts_manager.cc:194] Registered new tserver with Master: 628e7d55677e498ab92a9b1ac0ba055a (127.0.98.3:40757)
I20260926 01:11:35.006742 11617 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:56739
I20260926 01:11:35.008749 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 01:11:35.022410 11832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:11:35.014280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48280 (local address 127.0.98.62:43719)
0926 01:11:35.014328 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 01:11:35.014330 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:11:35.014702 (+ 372us) server_negotiation.cc:407] Connection header received
0926 01:11:35.014753 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:11:35.014755 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:11:35.014768 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:11:35.014846 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:11:35.015693 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:11:35.016478 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:11:35.017374 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:11:35.017659 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:11:35.017730 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:11:35.019342 (+ 1612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:11:35.019344 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:11:35.019344 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:11:35.019352 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:11:35.021025 (+ 1673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:11:35.021429 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:11:35.021431 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:11:35.021431 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:11:35.021558 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:11:35.021772 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:11:35.021773 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:11:35.021774 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:11:35.021954 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:11:35.022058 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 01:11:35.022208 (+ 150us) server_negotiation.cc:299] Negotiation successful
0926 01:11:35.022305 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 01:11:35.024586 11617 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48280:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:11:35.027050 11617 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:11:35.037246 12116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:11:35.030908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40757 (local address 127.0.0.1:49874)
0926 01:11:35.031994 (+ 1086us) negotiation.cc:107] Waiting for socket to connect
0926 01:11:35.032146 (+ 152us) client_negotiation.cc:174] Beginning negotiation
0926 01:11:35.032209 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:11:35.032609 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:11:35.032613 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:11:35.032616 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:11:35.032701 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:11:35.032703 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:11:35.034428 (+ 1725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:11:35.034429 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:11:35.035767 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:11:35.035771 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:11:35.036872 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:11:35.036873 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:11:35.037000 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:11:35.037017 (+ 17us) client_negotiation.cc:769] Sending connection context
0926 01:11:35.037087 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 01:11:35.037145 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1054,"thread_start_us":74,"threads_started":1}
I20260926 01:11:35.037256 12118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:11:35.031420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49874 (local address 127.0.98.3:40757)
0926 01:11:35.032442 (+ 1022us) server_negotiation.cc:206] Beginning negotiation
0926 01:11:35.032446 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 01:11:35.032455 (+ 9us) server_negotiation.cc:407] Connection header received
0926 01:11:35.032490 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:11:35.032492 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:11:35.032508 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:11:35.032575 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:11:35.032777 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:11:35.034340 (+ 1563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:11:35.035891 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:11:35.036764 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:11:35.036854 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:11:35.036945 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 01:11:35.037119 (+ 174us) server_negotiation.cc:299] Negotiation successful
0926 01:11:35.037166 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":989,"thread_start_us":75,"threads_started":1}
I20260926 01:11:35.038427 12035 tablet_service.cc:1467] Processing CreateTablet for tablet 3dd302c65d464a1b800addbd1b2941a0 (DEFAULT_TABLE table=test-table [id=a041d49ffca64d258142bcbca44b1426]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:11:35.038834 12035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3dd302c65d464a1b800addbd1b2941a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:11:35.041945 12119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:11:35.031594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39352 (local address 127.0.98.1:38867)
0926 01:11:35.032108 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0926 01:11:35.032113 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 01:11:35.037569 (+ 5456us) server_negotiation.cc:407] Connection header received
0926 01:11:35.037681 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:11:35.037684 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:11:35.037701 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:11:35.037770 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:11:35.038010 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:11:35.039389 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:11:35.040694 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:11:35.041446 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:11:35.041493 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:11:35.041549 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 01:11:35.041777 (+ 228us) server_negotiation.cc:299] Negotiation successful
0926 01:11:35.041860 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":480,"thread_start_us":51,"threads_started":1}
I20260926 01:11:35.042169 12116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:11:35.031146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38867 (local address 127.0.0.1:39352)
0926 01:11:35.037430 (+ 6284us) negotiation.cc:107] Waiting for socket to connect
0926 01:11:35.037441 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 01:11:35.037602 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:11:35.037906 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:11:35.037906 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:11:35.037908 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:11:35.037943 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:11:35.037945 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:11:35.039473 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:11:35.039473 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:11:35.040597 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:11:35.040600 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:11:35.041603 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:11:35.041603 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:11:35.041661 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:11:35.041664 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 01:11:35.042055 (+ 391us) client_negotiation.cc:240] Negotiation successful
0926 01:11:35.042105 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6271,"thread_start_us":57,"threads_started":1}
I20260926 01:11:35.042912 11763 tablet_service.cc:1467] Processing CreateTablet for tablet 3dd302c65d464a1b800addbd1b2941a0 (DEFAULT_TABLE table=test-table [id=a041d49ffca64d258142bcbca44b1426]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:11:35.043290 11763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3dd302c65d464a1b800addbd1b2941a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:11:35.046988 12121 tablet_bootstrap.cc:492] T 3dd302c65d464a1b800addbd1b2941a0 P 0624dd89dbdf48daaf5df4c0bc5f29da: Bootstrap starting.
I20260926 01:11:35.048161 12121 tablet_bootstrap.cc:654] T 3dd302c65d464a1b800addbd1b2941a0 P 0624dd89dbdf48daaf5df4c0bc5f29da: Neither blocks nor log segments found. Creating new log.
I20260926 01:11:35.048441 12121 log.cc:824] T 3dd302c65d464a1b800addbd1b2941a0 P 0624dd89dbdf48daaf5df4c0bc5f29da: Log is configured to *not* fsync() on all Append() calls
I20260926 01:11:35.048825 12120 tablet_bootstrap.cc:492] T 3dd302c65d464a1b800addbd1b2941a0 P 628e7d55677e498ab92a9b1ac0ba055a: Bootstrap starting.
I20260926 01:11:35.049899 12120 tablet_bootstrap.cc:654] T 3dd302c65d464a1b800addbd1b2941a0 P 628e7d55677e498ab92a9b1ac0ba055a: Neither blocks nor log segments found. Creating new log.
I20260926 01:11:35.050544 12120 log.cc:824] T 3dd302c65d464a1b800addbd1b2941a0 P 628e7d55677e498ab92a9b1ac0ba055a: Log is configured to *not* fsync() on all Append() calls
I20260926 01:11:35.051338 12121 tablet_bootstrap.cc:492] T 3dd302c65d464a1b800addbd1b2941a0 P 0624dd89dbdf48daaf5df4c0bc5f29da: No bootstrap required, opened a new log
I20260926 01:11:35.051412 12121 ts_tablet_manager.cc:1397] T 3dd302c65d464a1b800addbd1b2941a0 P 0624dd89dbdf48daaf5df4c0bc5f29da: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260926 01:11:35.054726 12120 tablet_bootstrap.cc:492] T 3dd302c65d464a1b800addbd1b2941a0 P 628e7d55677e498ab92a9b1ac0ba055a: No bootstrap required, opened a new log
I20260926 01:11:35.054808 12120 ts_tablet_manager.cc:1397] T 3dd302c65d464a1b800addbd1b2941a0 P 628e7d55677e498ab92a9b1ac0ba055a: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.001s
I20260926 01:11:35.057291 12120 raft_consensus.cc:348] T 3dd302c65d464a1b800addbd1b2941a0 P 628e7d55677e498ab92a9b1ac0ba055a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e09e749c5dd4e00ad5cea18e0080c41" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44623 } } peers { permanent_uuid: "628e7d55677e498ab92a9b1ac0ba055a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40757 } } peers { permanent_uuid: "0624dd89dbdf48daaf5df4c0bc5f29da" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38867 } }
I20260926 01:11:35.057485 12120 raft_consensus.cc:374] T 3dd302c65d464a1b800addbd1b2941a0 P 628e7d55677e498ab92a9b1ac0ba055a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:11:35.057518 12120 raft_consensus.cc:729] T 3dd302c65d464a1b800addbd1b2941a0 P 628e7d55677e498ab92a9b1ac0ba055a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 628e7d55677e498ab92a9b1ac0ba055a, State: Initialized, Role: FOLLOWER
I20260926 01:11:35.053974 12121 raft_consensus.cc:348] T 3dd302c65d464a1b800addbd1b2941a0 P 0624dd89dbdf48daaf5df4c0bc5f29da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e09e749c5dd4e00ad5cea18e0080c41" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44623 } } peers { permanent_uuid: "628e7d55677e498ab92a9b1ac0ba055a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40757 } } peers { permanent_uuid: "0624dd89dbdf48daaf5df4c0bc5f29da" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38867 } }
I20260926 01:11:35.057695 12121 raft_consensus.cc:374] T 3dd302c65d464a1b800addbd1b2941a0 P 0624dd89dbdf48daaf5df4c0bc5f29da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:11:35.057739 12121 raft_consensus.cc:729] T 3dd302c65d464a1b800addbd1b2941a0 P 0624dd89dbdf48daaf5df4c0bc5f29da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0624dd89dbdf48daaf5df4c0bc5f29da, State: Initialized, Role: FOLLOWER
I20260926 01:11:35.057897 12121 consensus_queue.cc:260] T 3dd302c65d464a1b800addbd1b2941a0 P 0624dd89dbdf48daaf5df4c0bc5f29da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e09e749c5dd4e00ad5cea18e0080c41" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44623 } } peers { permanent_uuid: "628e7d55677e498ab92a9b1ac0ba055a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40757 } } peers { permanent_uuid: "0624dd89dbdf48daaf5df4c0bc5f29da" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38867 } }
I20260926 01:11:35.058204 12121 ts_tablet_manager.cc:1428] T 3dd302c65d464a1b800addbd1b2941a0 P 0624dd89dbdf48daaf5df4c0bc5f29da: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20260926 01:11:35.058329 11829 heartbeater.cc:503] Master 127.0.98.62:43719 was elected leader, sending a full tablet report...
I20260926 01:11:35.058741 12114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:11:35.030910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44623 (local address 127.0.0.1:35360)
0926 01:11:35.032304 (+ 1394us) negotiation.cc:107] Waiting for socket to connect
0926 01:11:35.032312 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 01:11:35.032341 (+ 29us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:11:35.054174 (+ 21833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:11:35.054176 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:11:35.054179 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:11:35.054257 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:11:35.054259 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:11:35.055744 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:11:35.055745 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:11:35.056984 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:11:35.056989 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:11:35.058449 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:11:35.058450 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:11:35.058537 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:11:35.058543 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 01:11:35.058600 (+ 57us) client_negotiation.cc:240] Negotiation successful
0926 01:11:35.058656 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1378,"thread_start_us":96,"threads_started":1}
I20260926 01:11:35.059232 12117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:11:35.031184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35360 (local address 127.0.98.2:44623)
0926 01:11:35.053964 (+ 22780us) server_negotiation.cc:206] Beginning negotiation
0926 01:11:35.053969 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 01:11:35.053985 (+ 16us) server_negotiation.cc:407] Connection header received
0926 01:11:35.054031 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:11:35.054033 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:11:35.054052 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:11:35.054169 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:11:35.054359 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:11:35.055515 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:11:35.057135 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:11:35.057974 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:11:35.058063 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:11:35.058155 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 01:11:35.059099 (+ 944us) server_negotiation.cc:299] Negotiation successful
0926 01:11:35.059147 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22727,"thread_start_us":75,"threads_started":1}
I20260926 01:11:35.059343 12120 consensus_queue.cc:260] T 3dd302c65d464a1b800addbd1b2941a0 P 628e7d55677e498ab92a9b1ac0ba055a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e09e749c5dd4e00ad5cea18e0080c41" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44623 } } peers { permanent_uuid: "628e7d55677e498ab92a9b1ac0ba055a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40757 } } peers { permanent_uuid: "0624dd89dbdf48daaf5df4c0bc5f29da" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38867 } }
I20260926 01:11:35.059630 12120 ts_tablet_manager.cc:1428] T 3dd302c65d464a1b800addbd1b2941a0 P 628e7d55677e498ab92a9b1ac0ba055a: Time spent starting tablet: real 0.005s user 0.000s sys 0.003s
I20260926 01:11:35.059756 12101 heartbeater.cc:503] Master 127.0.98.62:43719 was elected leader, sending a full tablet report...
I20260926 01:11:35.059926 11900 tablet_service.cc:1467] Processing CreateTablet for tablet 3dd302c65d464a1b800addbd1b2941a0 (DEFAULT_TABLE table=test-table [id=a041d49ffca64d258142bcbca44b1426]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:11:35.060307 11900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3dd302c65d464a1b800addbd1b2941a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:11:35.063882 12126 tablet_bootstrap.cc:492] T 3dd302c65d464a1b800addbd1b2941a0 P 8e09e749c5dd4e00ad5cea18e0080c41: Bootstrap starting.
I20260926 01:11:35.064862 12126 tablet_bootstrap.cc:654] T 3dd302c65d464a1b800addbd1b2941a0 P 8e09e749c5dd4e00ad5cea18e0080c41: Neither blocks nor log segments found. Creating new log.
I20260926 01:11:35.065178 12126 log.cc:824] T 3dd302c65d464a1b800addbd1b2941a0 P 8e09e749c5dd4e00ad5cea18e0080c41: Log is configured to *not* fsync() on all Append() calls
I20260926 01:11:35.066450 12126 tablet_bootstrap.cc:492] T 3dd302c65d464a1b800addbd1b2941a0 P 8e09e749c5dd4e00ad5cea18e0080c41: No bootstrap required, opened a new log
I20260926 01:11:35.066540 12126 ts_tablet_manager.cc:1397] T 3dd302c65d464a1b800addbd1b2941a0 P 8e09e749c5dd4e00ad5cea18e0080c41: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 01:11:35.068526 12126 raft_consensus.cc:348] T 3dd302c65d464a1b800addbd1b2941a0 P 8e09e749c5dd4e00ad5cea18e0080c41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e09e749c5dd4e00ad5cea18e0080c41" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44623 } } peers { permanent_uuid: "628e7d55677e498ab92a9b1ac0ba055a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40757 } } peers { permanent_uuid: "0624dd89dbdf48daaf5df4c0bc5f29da" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38867 } }
I20260926 01:11:35.068714 12126 raft_consensus.cc:374] T 3dd302c65d464a1b800addbd1b2941a0 P 8e09e749c5dd4e00ad5cea18e0080c41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:11:35.068756 12126 raft_consensus.cc:729] T 3dd302c65d464a1b800addbd1b2941a0 P 8e09e749c5dd4e00ad5cea18e0080c41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e09e749c5dd4e00ad5cea18e0080c41, State: Initialized, Role: FOLLOWER
I20260926 01:11:35.068881 12126 consensus_queue.cc:260] T 3dd302c65d464a1b800addbd1b2941a0 P 8e09e749c5dd4e00ad5cea18e0080c41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e09e749c5dd4e00ad5cea18e0080c41" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44623 } } peers { permanent_uuid: "628e7d55677e498ab92a9b1ac0ba055a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40757 } } peers { permanent_uuid: "0624dd89dbdf48daaf5df4c0bc5f29da" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38867 } }
I20260926 01:11:35.069187 12126 ts_tablet_manager.cc:1428] T 3dd302c65d464a1b800addbd1b2941a0 P 8e09e749c5dd4e00ad5cea18e0080c41: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 01:11:35.069208 11966 heartbeater.cc:503] Master 127.0.98.62:43719 was elected leader, sending a full tablet report...
Sep 26 01:11:35 dist-test-slave-7b22 krb5kdc[11141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385094, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260926 01:11:35.205480 11967 tablet.cc:2369] T 3dd302c65d464a1b800addbd1b2941a0 P 8e09e749c5dd4e00ad5cea18e0080c41: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 01:11:35.225443 11830 tablet.cc:2369] T 3dd302c65d464a1b800addbd1b2941a0 P 0624dd89dbdf48daaf5df4c0bc5f29da: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 01:11:35.244233 12102 tablet.cc:2369] T 3dd302c65d464a1b800addbd1b2941a0 P 628e7d55677e498ab92a9b1ac0ba055a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 01:11:35.297025 12125 raft_consensus.cc:482] T 3dd302c65d464a1b800addbd1b2941a0 P 628e7d55677e498ab92a9b1ac0ba055a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:11:35.297183 12125 raft_consensus.cc:504] T 3dd302c65d464a1b800addbd1b2941a0 P 628e7d55677e498ab92a9b1ac0ba055a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e09e749c5dd4e00ad5cea18e0080c41" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44623 } } peers { permanent_uuid: "628e7d55677e498ab92a9b1ac0ba055a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40757 } } peers { permanent_uuid: "0624dd89dbdf48daaf5df4c0bc5f29da" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38867 } }
I20260926 01:11:35.297593 12125 leader_election.cc:288] T 3dd302c65d464a1b800addbd1b2941a0 P 628e7d55677e498ab92a9b1ac0ba055a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8e09e749c5dd4e00ad5cea18e0080c41 (127.0.98.2:44623), 0624dd89dbdf48daaf5df4c0bc5f29da (127.0.98.1:38867)
I20260926 01:11:35.301725 12103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:11:35.297787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44623 (local address 127.0.98.3:55723)
0926 01:11:35.297919 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0926 01:11:35.297931 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 01:11:35.297985 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:11:35.298320 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:11:35.298321 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:11:35.298324 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:11:35.298413 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:11:35.298415 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:11:35.299459 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:11:35.299460 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:11:35.300700 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:11:35.300703 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:11:35.301481 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:11:35.301482 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:11:35.301554 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:11:35.301557 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 01:11:35.301601 (+ 44us) client_negotiation.cc:240] Negotiation successful
0926 01:11:35.301643 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"mutex_wait_us":41}
I20260926 01:11:35.301950 12119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:11:35.298079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46791 (local address 127.0.98.1:38867)
0926 01:11:35.298129 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 01:11:35.298132 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:11:35.298483 (+ 351us) server_negotiation.cc:407] Connection header received
0926 01:11:35.298510 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:11:35.298511 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:11:35.298521 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:11:35.298569 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:11:35.298758 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:11:35.299546 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:11:35.300953 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:11:35.301531 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:11:35.301581 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:11:35.301591 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 01:11:35.301808 (+ 217us) server_negotiation.cc:299] Negotiation successful
0926 01:11:35.301878 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260926 01:11:35.302175 12129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:11:35.297807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38867 (local address 127.0.98.3:46791)
0926 01:11:35.298228 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0926 01:11:35.298237 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 01:11:35.298287 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:11:35.298586 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:11:35.298587 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:11:35.298588 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:11:35.298661 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:11:35.298663 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:11:35.299643 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:11:35.299644 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:11:35.300849 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:11:35.300852 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:11:35.301651 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:11:35.301652 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:11:35.301732 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:11:35.301736 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 01:11:35.302029 (+ 293us) client_negotiation.cc:240] Negotiation successful
0926 01:11:35.302087 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":189,"threads_started":1}
I20260926 01:11:35.302446 12117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:11:35.298139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55723 (local address 127.0.98.2:44623)
0926 01:11:35.298185 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 01:11:35.298187 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:11:35.298196 (+ 9us) server_negotiation.cc:407] Connection header received
0926 01:11:35.298227 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:11:35.298228 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:11:35.298239 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:11:35.298291 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:11:35.298616 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:11:35.299355 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:11:35.300806 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:11:35.301355 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:11:35.301436 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:11:35.301450 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0926 01:11:35.302342 (+ 892us) server_negotiation.cc:299] Negotiation successful
0926 01:11:35.302385 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 01:11:35.302567 11765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dd302c65d464a1b800addbd1b2941a0" candidate_uuid: "628e7d55677e498ab92a9b1ac0ba055a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0624dd89dbdf48daaf5df4c0bc5f29da" is_pre_election: true
I20260926 01:11:35.302722 11765 raft_consensus.cc:2454] T 3dd302c65d464a1b800addbd1b2941a0 P 0624dd89dbdf48daaf5df4c0bc5f29da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 628e7d55677e498ab92a9b1ac0ba055a in term 0.
I20260926 01:11:35.302690 11920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dd302c65d464a1b800addbd1b2941a0" candidate_uuid: "628e7d55677e498ab92a9b1ac0ba055a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e09e749c5dd4e00ad5cea18e0080c41" is_pre_election: true
I20260926 01:11:35.302842 11920 raft_consensus.cc:2454] T 3dd302c65d464a1b800addbd1b2941a0 P 8e09e749c5dd4e00ad5cea18e0080c41 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 628e7d55677e498ab92a9b1ac0ba055a in term 0.
I20260926 01:11:35.302934 11990 leader_election.cc:302] T 3dd302c65d464a1b800addbd1b2941a0 P 628e7d55677e498ab92a9b1ac0ba055a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0624dd89dbdf48daaf5df4c0bc5f29da, 628e7d55677e498ab92a9b1ac0ba055a; no voters:
I20260926 01:11:35.303124 12125 raft_consensus.cc:2781] T 3dd302c65d464a1b800addbd1b2941a0 P 628e7d55677e498ab92a9b1ac0ba055a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:11:35.303171 12125 raft_consensus.cc:482] T 3dd302c65d464a1b800addbd1b2941a0 P 628e7d55677e498ab92a9b1ac0ba055a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:11:35.303190 12125 raft_consensus.cc:3037] T 3dd302c65d464a1b800addbd1b2941a0 P 628e7d55677e498ab92a9b1ac0ba055a [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:11:35.304580 12125 raft_consensus.cc:504] T 3dd302c65d464a1b800addbd1b2941a0 P 628e7d55677e498ab92a9b1ac0ba055a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e09e749c5dd4e00ad5cea18e0080c41" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44623 } } peers { permanent_uuid: "628e7d55677e498ab92a9b1ac0ba055a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40757 } } peers { permanent_uuid: "0624dd89dbdf48daaf5df4c0bc5f29da" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38867 } }
I20260926 01:11:35.304723 12125 leader_election.cc:288] T 3dd302c65d464a1b800addbd1b2941a0 P 628e7d55677e498ab92a9b1ac0ba055a [CANDIDATE]: Term 1 election: Requested vote from peers 8e09e749c5dd4e00ad5cea18e0080c41 (127.0.98.2:44623), 0624dd89dbdf48daaf5df4c0bc5f29da (127.0.98.1:38867)
I20260926 01:11:35.304949 11765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dd302c65d464a1b800addbd1b2941a0" candidate_uuid: "628e7d55677e498ab92a9b1ac0ba055a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0624dd89dbdf48daaf5df4c0bc5f29da"
I20260926 01:11:35.304955 11920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dd302c65d464a1b800addbd1b2941a0" candidate_uuid: "628e7d55677e498ab92a9b1ac0ba055a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e09e749c5dd4e00ad5cea18e0080c41"
I20260926 01:11:35.305024 11765 raft_consensus.cc:3037] T 3dd302c65d464a1b800addbd1b2941a0 P 0624dd89dbdf48daaf5df4c0bc5f29da [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:11:35.305035 11920 raft_consensus.cc:3037] T 3dd302c65d464a1b800addbd1b2941a0 P 8e09e749c5dd4e00ad5cea18e0080c41 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:11:35.305923 11920 raft_consensus.cc:2454] T 3dd302c65d464a1b800addbd1b2941a0 P 8e09e749c5dd4e00ad5cea18e0080c41 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 628e7d55677e498ab92a9b1ac0ba055a in term 1.
I20260926 01:11:35.305963 11765 raft_consensus.cc:2454] T 3dd302c65d464a1b800addbd1b2941a0 P 0624dd89dbdf48daaf5df4c0bc5f29da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 628e7d55677e498ab92a9b1ac0ba055a in term 1.
I20260926 01:11:35.306134 11991 leader_election.cc:302] T 3dd302c65d464a1b800addbd1b2941a0 P 628e7d55677e498ab92a9b1ac0ba055a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 628e7d55677e498ab92a9b1ac0ba055a, 8e09e749c5dd4e00ad5cea18e0080c41; no voters:
I20260926 01:11:35.306221 12125 raft_consensus.cc:2781] T 3dd302c65d464a1b800addbd1b2941a0 P 628e7d55677e498ab92a9b1ac0ba055a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:11:35.306378 12125 raft_consensus.cc:686] T 3dd302c65d464a1b800addbd1b2941a0 P 628e7d55677e498ab92a9b1ac0ba055a [term 1 LEADER]: Becoming Leader. State: Replica: 628e7d55677e498ab92a9b1ac0ba055a, State: Running, Role: LEADER
I20260926 01:11:35.306483 12125 consensus_queue.cc:237] T 3dd302c65d464a1b800addbd1b2941a0 P 628e7d55677e498ab92a9b1ac0ba055a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e09e749c5dd4e00ad5cea18e0080c41" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44623 } } peers { permanent_uuid: "628e7d55677e498ab92a9b1ac0ba055a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40757 } } peers { permanent_uuid: "0624dd89dbdf48daaf5df4c0bc5f29da" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38867 } }
I20260926 01:11:35.307257 11617 catalog_manager.cc:5495] T 3dd302c65d464a1b800addbd1b2941a0 P 628e7d55677e498ab92a9b1ac0ba055a reported cstate change: term changed from 0 to 1, leader changed from <none> to 628e7d55677e498ab92a9b1ac0ba055a (127.0.98.3). New cstate: current_term: 1 leader_uuid: "628e7d55677e498ab92a9b1ac0ba055a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e09e749c5dd4e00ad5cea18e0080c41" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44623 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "628e7d55677e498ab92a9b1ac0ba055a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40757 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0624dd89dbdf48daaf5df4c0bc5f29da" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38867 } health_report { overall_health: UNKNOWN } } }
I20260926 01:11:35.321465 12118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:11:35.317672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49890 (local address 127.0.98.3:40757)
0926 01:11:35.317718 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 01:11:35.317720 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:11:35.317731 (+ 11us) server_negotiation.cc:407] Connection header received
0926 01:11:35.317850 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:11:35.317851 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:11:35.317876 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:11:35.317950 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:11:35.318140 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:11:35.318873 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:11:35.320170 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:11:35.320461 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:11:35.320595 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:11:35.320858 (+ 263us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:11:35.321039 (+ 181us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:11:35.321129 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 01:11:35.321267 (+ 138us) server_negotiation.cc:299] Negotiation successful
0926 01:11:35.321301 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 01:11:35.323695 11765 raft_consensus.cc:1261] T 3dd302c65d464a1b800addbd1b2941a0 P 0624dd89dbdf48daaf5df4c0bc5f29da [term 1 FOLLOWER]: Refusing update from remote peer 628e7d55677e498ab92a9b1ac0ba055a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:11:35.323791 11920 raft_consensus.cc:1261] T 3dd302c65d464a1b800addbd1b2941a0 P 8e09e749c5dd4e00ad5cea18e0080c41 [term 1 FOLLOWER]: Refusing update from remote peer 628e7d55677e498ab92a9b1ac0ba055a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:11:35.324321 12130 consensus_queue.cc:1035] T 3dd302c65d464a1b800addbd1b2941a0 P 628e7d55677e498ab92a9b1ac0ba055a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0624dd89dbdf48daaf5df4c0bc5f29da" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:11:35.324529 12130 consensus_queue.cc:1035] T 3dd302c65d464a1b800addbd1b2941a0 P 628e7d55677e498ab92a9b1ac0ba055a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e09e749c5dd4e00ad5cea18e0080c41" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:11:35.327847 12133 mvcc.cc:205] Tried to move back new op lower bound from 7333417350442745856 to 7333417350375735296. Current Snapshot: MvccSnapshot[applied={T|T < 7333417350442745856}]
I20260926 01:11:35.328964 12135 mvcc.cc:205] Tried to move back new op lower bound from 7333417350442745856 to 7333417350375735296. Current Snapshot: MvccSnapshot[applied={T|T < 7333417350442745856}]
I20260926 01:11:35.331251 12134 mvcc.cc:205] Tried to move back new op lower bound from 7333417350442745856 to 7333417350375735296. Current Snapshot: MvccSnapshot[applied={T|T < 7333417350442745856}]
I20260926 01:11:35.332455 11616 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48280:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:11:35.332552 11616 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48280:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:11:35.333371 11616 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3541b67264cf4dafb3ea63b68bb28c08: Sending DeleteTablet for 3 replicas of tablet 3dd302c65d464a1b800addbd1b2941a0
I20260926 01:11:35.334125 11900 tablet_service.cc:1514] Processing DeleteTablet for tablet 3dd302c65d464a1b800addbd1b2941a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:11:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35360
I20260926 01:11:35.334342 11763 tablet_service.cc:1514] Processing DeleteTablet for tablet 3dd302c65d464a1b800addbd1b2941a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:11:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39352
I20260926 01:11:35.334687 12035 tablet_service.cc:1514] Processing DeleteTablet for tablet 3dd302c65d464a1b800addbd1b2941a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:11:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49874
I20260926 01:11:35.335246 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 11698
I20260926 01:11:35.339488 12142 tablet_replica.cc:330] T 3dd302c65d464a1b800addbd1b2941a0 P 8e09e749c5dd4e00ad5cea18e0080c41: stopping tablet replica
I20260926 01:11:35.339653 12144 tablet_replica.cc:330] T 3dd302c65d464a1b800addbd1b2941a0 P 628e7d55677e498ab92a9b1ac0ba055a: stopping tablet replica
I20260926 01:11:35.340072 12144 raft_consensus.cc:2229] T 3dd302c65d464a1b800addbd1b2941a0 P 628e7d55677e498ab92a9b1ac0ba055a [term 1 LEADER]: Raft consensus shutting down.
I20260926 01:11:35.340240 12142 raft_consensus.cc:2229] T 3dd302c65d464a1b800addbd1b2941a0 P 8e09e749c5dd4e00ad5cea18e0080c41 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:11:35.340407 12144 raft_consensus.cc:2258] T 3dd302c65d464a1b800addbd1b2941a0 P 628e7d55677e498ab92a9b1ac0ba055a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:11:35.340575 12142 raft_consensus.cc:2258] T 3dd302c65d464a1b800addbd1b2941a0 P 8e09e749c5dd4e00ad5cea18e0080c41 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:11:35.342192 12144 ts_tablet_manager.cc:1905] T 3dd302c65d464a1b800addbd1b2941a0 P 628e7d55677e498ab92a9b1ac0ba055a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:11:35.342687 12142 ts_tablet_manager.cc:1905] T 3dd302c65d464a1b800addbd1b2941a0 P 8e09e749c5dd4e00ad5cea18e0080c41: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:11:35.346451 12142 ts_tablet_manager.cc:1918] T 3dd302c65d464a1b800addbd1b2941a0 P 8e09e749c5dd4e00ad5cea18e0080c41: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:11:35.346628 12142 log.cc:1196] T 3dd302c65d464a1b800addbd1b2941a0 P 8e09e749c5dd4e00ad5cea18e0080c41: Deleting WAL directory at /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384993574762-392-0/minicluster-data/ts-1/wal/wals/3dd302c65d464a1b800addbd1b2941a0
I20260926 01:11:35.346815 12144 ts_tablet_manager.cc:1918] T 3dd302c65d464a1b800addbd1b2941a0 P 628e7d55677e498ab92a9b1ac0ba055a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:11:35.346899 12144 log.cc:1196] T 3dd302c65d464a1b800addbd1b2941a0 P 628e7d55677e498ab92a9b1ac0ba055a: Deleting WAL directory at /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384993574762-392-0/minicluster-data/ts-2/wal/wals/3dd302c65d464a1b800addbd1b2941a0
I20260926 01:11:35.347110 12142 ts_tablet_manager.cc:1939] T 3dd302c65d464a1b800addbd1b2941a0 P 8e09e749c5dd4e00ad5cea18e0080c41: Deleting consensus metadata
I20260926 01:11:35.347175 12144 ts_tablet_manager.cc:1939] T 3dd302c65d464a1b800addbd1b2941a0 P 628e7d55677e498ab92a9b1ac0ba055a: Deleting consensus metadata
I20260926 01:11:35.347628 11602 catalog_manager.cc:4841] TS 628e7d55677e498ab92a9b1ac0ba055a (127.0.98.3:40757): tablet 3dd302c65d464a1b800addbd1b2941a0 (table test-table [id=a041d49ffca64d258142bcbca44b1426]) successfully deleted
I20260926 01:11:35.347751 11604 catalog_manager.cc:4841] TS 8e09e749c5dd4e00ad5cea18e0080c41 (127.0.98.2:44623): tablet 3dd302c65d464a1b800addbd1b2941a0 (table test-table [id=a041d49ffca64d258142bcbca44b1426]) successfully deleted
W20260926 01:11:35.347939 11603 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38867 (error 108)
I20260926 01:11:35.348857 12114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:11:35.348741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38867 (local address 127.0.0.1:39354)
0926 01:11:35.348790 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0926 01:11:35.348822 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260926 01:11:35.349104 11603 catalog_manager.cc:4584] TS 0624dd89dbdf48daaf5df4c0bc5f29da (127.0.98.1:38867): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3dd302c65d464a1b800addbd1b2941a0: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38867: connect: Connection refused (error 111)
I20260926 01:11:35.353279 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 11835
I20260926 01:11:35.359581 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 11972
I20260926 01:11:35.366010 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 11585
I20260926 01:11:35.372480 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260926 01:11:35.707170 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260926 01:11:35.707286 392 mini_ranger.cc:66] Stopping Ranger...
I20260926 01:11:36.042266 392 mini_ranger.cc:68] Stopped Ranger
2026-09-26 01:11:36.042 UTC [11171] LOG: received smart shutdown request
2026-09-26 01:11:36.045 UTC [11171] LOG: background worker "logical replication launcher" (PID 11180) exited with exit code 1
2026-09-26 01:11:36.046 UTC [11175] LOG: shutting down
2026-09-26 01:11:36.219 UTC [11171] LOG: database system is shut down
2026-09-26T01:11:36Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (46391 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 01:11:36 dist-test-slave-7b22 krb5kdc[12155](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 01:11:36 dist-test-slave-7b22 krb5kdc[12155](info): set up 2 sockets
Sep 26 01:11:36 dist-test-slave-7b22 krb5kdc[12155](info): commencing operation
krb5kdc: starting...
W20260926 01:11:37.249782 392 mini_kdc.cc:121] Time spent starting KDC: real 0.926s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/krb5kdc/test-user.keytab.
Sep 26 01:11:37 dist-test-slave-7b22 krb5kdc[12155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385097, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T01:11:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:11:37Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260926 01:11:37.417178 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskzzHEKz/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/postgres -l logfile start
2026-09-26 01:11:40.086 UTC [12185] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-26 01:11:40.086 UTC [12185] LOG: listening on IPv4 address "127.0.98.20", port 43107
2026-09-26 01:11:40.090 UTC [12185] LOG: listening on Unix socket "/tmp/.s.PGSQL.43107"
2026-09-26 01:11:40.110 UTC [12188] LOG: database system was shut down at 2026-09-26 01:11:38 UTC
2026-09-26 01:11:40.114 UTC [12185] LOG: database system is ready to accept connections
I20260926 01:11:40.989488 392 mini_postgres.cc:96] Postgres bound to 43107
2026-09-26 01:11:40.995 UTC [12196] FATAL: database "slave" does not exist
127.0.98.20:43107 - accepting connections
I20260926 01:11:40.995787 392 mini_ranger.cc:161] Starting Ranger...
I20260926 01:11:41.010735 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260926 01:11:41.529065 392 mini_ranger.cc:87] Created ranger Postgres database
I20260926 01:11:41.529162 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/ranger-admin
2026-09-26 01:11:41,804 [I] DB FLAVOR :POSTGRES
2026-09-26 01:11:41,805 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 01:11:41,805 [I] Checking connection..
2026-09-26 01:11:41,805 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzzHEKz/build/release/bin/postgresql.jar:/tmp/dist-test-taskzzHEKz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43107/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-26 01:11:42,068 [I] Checking connection passed.
2026-09-26 01:11:42,068 [I] --------- Verifying version history table ---------
2026-09-26 01:11:42,068 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzzHEKz/build/release/bin/postgresql.jar:/tmp/dist-test-taskzzHEKz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43107/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 01:11:42,315 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 01:11:42,315 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzzHEKz/build/release/bin/postgresql.jar:/tmp/dist-test-taskzzHEKz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43107/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 01:11:42,555 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 01:11:42,556 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-26 01:11:42,556 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzzHEKz/build/release/bin/postgresql.jar:/tmp/dist-test-taskzzHEKz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43107/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskzzHEKz/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-26 01:11:42.818 UTC [12260] WARNING: there is no transaction in progress
2026-09-26 01:11:42,824 [I] create_dbversion_catalog.sql file imported successfully
2026-09-26 01:11:42,824 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzzHEKz/build/release/bin/postgresql.jar:/tmp/dist-test-taskzzHEKz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43107/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 01:11:43,075 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-26 01:11:43,075 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 01:11:43,076 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzzHEKz/build/release/bin/postgresql.jar:/tmp/dist-test-taskzzHEKz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43107/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-26 01:11:43,316 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzzHEKz/build/release/bin/postgresql.jar:/tmp/dist-test-taskzzHEKz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43107/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-26 01:11:43,564 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzzHEKz/build/release/bin/postgresql.jar:/tmp/dist-test-taskzzHEKz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43107/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-7b22.c.gcp-upstream.internal','N') ;"
2026-09-26 01:11:43,801 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzzHEKz/build/release/bin/postgresql.jar:/tmp/dist-test-taskzzHEKz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43107/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 01:11:44,051 [I] Table x_portal_user does not exist in database ranger
2026-09-26 01:11:44,051 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzzHEKz/build/release/bin/postgresql.jar:/tmp/dist-test-taskzzHEKz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43107/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 01:11:44,304 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-26 01:11:44,305 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzzHEKz/build/release/bin/postgresql.jar:/tmp/dist-test-taskzzHEKz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43107/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 01:11:44,541 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzzHEKz/build/release/bin/postgresql.jar:/tmp/dist-test-taskzzHEKz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43107/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 01:11:44,778 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-26 01:11:44,778 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzzHEKz/build/release/bin/postgresql.jar:/tmp/dist-test-taskzzHEKz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43107/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskzzHEKz/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-26 01:11:45.835 UTC [12368] WARNING: there is no transaction in progress
2026-09-26 01:11:45.853 UTC [12368] WARNING: there is no transaction in progress
2026-09-26 01:11:45.871 UTC [12368] WARNING: there is no transaction in progress
2026-09-26 01:11:45.888 UTC [12368] WARNING: there is no transaction in progress
2026-09-26 01:11:45.908 UTC [12368] WARNING: there is no transaction in progress
2026-09-26 01:11:45.931 UTC [12368] WARNING: there is no transaction in progress
2026-09-26 01:11:46.000 UTC [12368] WARNING: there is no transaction in progress
2026-09-26 01:11:46.008 UTC [12368] WARNING: there is no transaction in progress
2026-09-26 01:11:46.025 UTC [12368] WARNING: there is no transaction in progress
2026-09-26 01:11:46.036 UTC [12368] WARNING: there is no transaction in progress
2026-09-26 01:11:46.047 UTC [12368] WARNING: there is no transaction in progress
2026-09-26 01:11:46.062 UTC [12368] WARNING: there is no transaction in progress
2026-09-26 01:11:46.073 UTC [12368] WARNING: there is no transaction in progress
2026-09-26 01:11:46.082 UTC [12368] WARNING: there is no transaction in progress
2026-09-26 01:11:46.091 UTC [12368] WARNING: there is no transaction in progress
2026-09-26 01:11:46.100 UTC [12368] WARNING: there is no transaction in progress
2026-09-26 01:11:46,816 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-26 01:11:46,817 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzzHEKz/build/release/bin/postgresql.jar:/tmp/dist-test-taskzzHEKz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43107/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-26 01:11:47,075 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-26 01:11:47,076 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzzHEKz/build/release/bin/postgresql.jar:/tmp/dist-test-taskzzHEKz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43107/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 01:11:47,328 [I] Table x_portal_user already exists in database 'ranger'
2026-09-26 01:11:47,329 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzzHEKz/build/release/bin/postgresql.jar:/tmp/dist-test-taskzzHEKz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43107/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 01:11:47,605 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-26 01:11:47,606 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzzHEKz/build/release/bin/postgresql.jar:/tmp/dist-test-taskzzHEKz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43107/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 01:11:47,842 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzzHEKz/build/release/bin/postgresql.jar:/tmp/dist-test-taskzzHEKz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43107/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 01:11:48,104 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzzHEKz/build/release/bin/postgresql.jar:/tmp/dist-test-taskzzHEKz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43107/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-7b22.c.gcp-upstream.internal';"
2026-09-26 01:11:48,342 [I] CORE_DB_SCHEMA import status has been updated
2026-09-26 01:11:48,343 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzzHEKz/build/release/bin/postgresql.jar:/tmp/dist-test-taskzzHEKz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43107/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-26 01:11:48,608 [I] DB_PATCHES have already been applied
I20260926 01:11:48.614009 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskzzHEKz/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskzzHEKz/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskzzHEKz/build/release/bin/postgresql.jar:/tmp/dist-test-taskzzHEKz/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260926 01:11:48.614084 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260926 01:11:48.617102 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:44599
Sep 26, 2026 1:11:48 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 1:11:48 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 1:11:49 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 26, 2026 1:11:49 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 26, 2026 1:11:49 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 26, 2026 1:11:49 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 26, 2026 1:11:49 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260926 01:11:49.615720 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.620s user 0.000s sys 0.008s
Sep 26, 2026 1:12:10 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 26, 2026 1:12:10 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 26, 2026 1:12:10 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 26, 2026 1:12:10 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 26, 2026 1:12:10 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 26, 2026 1:12:10 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 26, 2026 1:12:10 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 26, 2026 1:12:10 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 26, 2026 1:12:10 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 26, 2026 1:12:10 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 26, 2026 1:12:10 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 26, 2026 1:12:10 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 26, 2026 1:12:10 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 26, 2026 1:12:10 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 26, 2026 1:12:10 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 26, 2026 1:12:10 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 26, 2026 1:12:10 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 26, 2026 1:12:11 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260926 01:12:12.514998 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 26 01:12:12 dist-test-slave-7b22 krb5kdc[12155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385132, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260926 01:12:12.608814 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260926 01:12:12.618093 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260926 01:12:13.587241 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260926 01:12:13.587329 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/ranger-kms
I20260926 01:12:13.587347 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:43107
2026-09-26 01:12:13,772 [I] DB FLAVOR :POSTGRES
2026-09-26 01:12:13,772 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 01:12:13,772 [I] Checking connection
2026-09-26 01:12:13,772 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzzHEKz/build/release/bin/postgresql.jar:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43107/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-26 01:12:14,044 [I] connection success
2026-09-26 01:12:14,045 [I] --------- Verifying Ranger DB tables ---------
2026-09-26 01:12:14,045 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-26 01:12:14,045 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzzHEKz/build/release/bin/postgresql.jar:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43107/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-26 01:12:14,302 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-26 01:12:14,303 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 01:12:14,303 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-26 01:12:14,303 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzzHEKz/build/release/bin/postgresql.jar:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43107/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-26 01:12:14,557 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260926 01:12:15.050364 392 mini_ranger_kms.cc:310] Created kms service
I20260926 01:12:15.235163 392 mini_ranger_kms.cc:326] Created kudu user
I20260926 01:12:15.286772 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 26 01:12:15 dist-test-slave-7b22 krb5kdc[12155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385132, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 01:12:15.775120 392 mini_ranger_kms.cc:384] Added ranger policy
I20260926 01:12:15.775235 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskzzHEKz/build/release/bin/postgresql.jar:/tmp/dist-test-taskzzHEKz/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskzzHEKz/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260926 01:12:15.775252 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260926 01:12:15.778628 392 mini_ranger_kms.cc:276] Ranger KMS PID: 12549
I20260926 01:12:15.778713 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:38041
Sep 26, 2026 1:12:16 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 1:12:16 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 1:12:16 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 26, 2026 1:12:16 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 26, 2026 1:12:17 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 26, 2026 1:12:17 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260926 01:12:17.869154 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.260s user 0.009s sys 0.159s
I20260926 01:12:17.869266 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260926 01:12:17.869313 392 mini_ranger_kms.cc:401] 127.0.98.20:38041/kms/v1/keys
Sep 26 01:12:17 dist-test-slave-7b22 krb5kdc[12155](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790385137, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 26 01:12:18 dist-test-slave-7b22 krb5kdc[12155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385137, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 26 01:12:18 dist-test-slave-7b22 krb5kdc[12155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385137, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 26 01:12:21 dist-test-slave-7b22 krb5kdc[12155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385141, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 01:12:21.196385 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43267
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43267
--ranger_config_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:38041/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:12:21.277731 12599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:12:21.278020 12599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:12:21.278043 12599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:12:21.279284 12599 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:12:21.279331 12599 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:12:21.279345 12599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:12:21.279356 12599 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:12:21.279366 12599 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:12:21.281064 12599 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:38041/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43267
--ranger_config_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43267
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12599
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:12:21.281293 12599 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:12:21.281514 12599 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:12:21.286245 12607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:12:21.286245 12605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:12:21.286283 12604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:12:21.287212 12599 server_base.cc:1034] running on GCE node
I20260926 01:12:21.287477 12599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:12:21.287746 12599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:12:21.288899 12599 hybrid_clock.cc:648] HybridClock initialized: now 1790385141288886 us; error 32 us; skew 500 ppm
Sep 26 01:12:21 dist-test-slave-7b22 krb5kdc[12155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385141, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:12:21.291766 12599 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 01:12:21.292300 12599 webserver.cc:460] Webserver started at http://127.0.98.62:45621/ using document root <none> and password file <none>
I20260926 01:12:21.292500 12599 fs_manager.cc:359] Metadata directory not provided
I20260926 01:12:21.292546 12599 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:12:21.292675 12599 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 01:12:21 dist-test-slave-7b22 krb5kdc[12155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385141, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 01:12:21.341291 12599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/master-0/data/instance:
uuid: "f82caa48b6ce457880e82369fd70ed96"
format_stamp: "Formatted at 2026-09-26 01:12:21 on dist-test-slave-7b22"
server_key: "d44b7968ead54ca8d4b4b02b93d1f594"
server_key_iv: "f9d511ec6a8d6186e39a9da822b683a3"
server_key_version: "kuduclusterkey@0"
I20260926 01:12:21.341714 12599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/master-0/wal/instance:
uuid: "f82caa48b6ce457880e82369fd70ed96"
format_stamp: "Formatted at 2026-09-26 01:12:21 on dist-test-slave-7b22"
server_key: "d44b7968ead54ca8d4b4b02b93d1f594"
server_key_iv: "f9d511ec6a8d6186e39a9da822b683a3"
server_key_version: "kuduclusterkey@0"
I20260926 01:12:21.343104 12599 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 01:12:21.355234 12614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:12:21.355438 12599 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 01:12:21.355530 12599 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/master-0/wal
uuid: "f82caa48b6ce457880e82369fd70ed96"
format_stamp: "Formatted at 2026-09-26 01:12:21 on dist-test-slave-7b22"
server_key: "d44b7968ead54ca8d4b4b02b93d1f594"
server_key_iv: "f9d511ec6a8d6186e39a9da822b683a3"
server_key_version: "kuduclusterkey@0"
I20260926 01:12:21.355626 12599 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:12:21.367202 12599 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:12:21.369110 12599 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:12:21.369256 12599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:12:21.373586 12599 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43267
I20260926 01:12:21.373628 12666 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43267 every 8 connection(s)
I20260926 01:12:21.374025 12599 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
I20260926 01:12:21.374794 12667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:12:21.377466 12667 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f82caa48b6ce457880e82369fd70ed96: Bootstrap starting.
I20260926 01:12:21.378017 12667 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f82caa48b6ce457880e82369fd70ed96: Neither blocks nor log segments found. Creating new log.
I20260926 01:12:21.378244 12667 log.cc:824] T 00000000000000000000000000000000 P f82caa48b6ce457880e82369fd70ed96: Log is configured to *not* fsync() on all Append() calls
I20260926 01:12:21.378829 12667 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f82caa48b6ce457880e82369fd70ed96: No bootstrap required, opened a new log
I20260926 01:12:21.380833 12667 raft_consensus.cc:348] T 00000000000000000000000000000000 P f82caa48b6ce457880e82369fd70ed96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f82caa48b6ce457880e82369fd70ed96" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43267 } }
I20260926 01:12:21.380986 12667 raft_consensus.cc:374] T 00000000000000000000000000000000 P f82caa48b6ce457880e82369fd70ed96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:12:21.381008 12667 raft_consensus.cc:729] T 00000000000000000000000000000000 P f82caa48b6ce457880e82369fd70ed96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f82caa48b6ce457880e82369fd70ed96, State: Initialized, Role: FOLLOWER
I20260926 01:12:21.381109 12667 consensus_queue.cc:260] T 00000000000000000000000000000000 P f82caa48b6ce457880e82369fd70ed96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f82caa48b6ce457880e82369fd70ed96" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43267 } }
I20260926 01:12:21.381163 12667 raft_consensus.cc:388] T 00000000000000000000000000000000 P f82caa48b6ce457880e82369fd70ed96 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:12:21.381188 12667 raft_consensus.cc:482] T 00000000000000000000000000000000 P f82caa48b6ce457880e82369fd70ed96 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:12:21.381225 12667 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f82caa48b6ce457880e82369fd70ed96 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:12:21.381757 12667 raft_consensus.cc:504] T 00000000000000000000000000000000 P f82caa48b6ce457880e82369fd70ed96 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f82caa48b6ce457880e82369fd70ed96" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43267 } }
I20260926 01:12:21.381891 12667 leader_election.cc:302] T 00000000000000000000000000000000 P f82caa48b6ce457880e82369fd70ed96 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f82caa48b6ce457880e82369fd70ed96; no voters:
I20260926 01:12:21.382040 12667 leader_election.cc:288] T 00000000000000000000000000000000 P f82caa48b6ce457880e82369fd70ed96 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:12:21.382077 12670 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f82caa48b6ce457880e82369fd70ed96 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:12:21.382277 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 12599
I20260926 01:12:21.382305 12670 raft_consensus.cc:686] T 00000000000000000000000000000000 P f82caa48b6ce457880e82369fd70ed96 [term 1 LEADER]: Becoming Leader. State: Replica: f82caa48b6ce457880e82369fd70ed96, State: Running, Role: LEADER
I20260926 01:12:21.382375 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/master-0/wal/instance
I20260926 01:12:21.382408 12670 consensus_queue.cc:237] T 00000000000000000000000000000000 P f82caa48b6ce457880e82369fd70ed96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f82caa48b6ce457880e82369fd70ed96" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43267 } }
I20260926 01:12:21.382314 12667 sys_catalog.cc:564] T 00000000000000000000000000000000 P f82caa48b6ce457880e82369fd70ed96 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:12:21.382560 392 external_mini_cluster.cc:1428] Setting key fe615342c0ff6682fe9e9a01b9fbdfbe
I20260926 01:12:21.382807 12671 sys_catalog.cc:455] T 00000000000000000000000000000000 P f82caa48b6ce457880e82369fd70ed96 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f82caa48b6ce457880e82369fd70ed96" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f82caa48b6ce457880e82369fd70ed96" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43267 } } }
I20260926 01:12:21.382884 12671 sys_catalog.cc:458] T 00000000000000000000000000000000 P f82caa48b6ce457880e82369fd70ed96 [sys.catalog]: This master's current role is: LEADER
I20260926 01:12:21.383405 12672 sys_catalog.cc:455] T 00000000000000000000000000000000 P f82caa48b6ce457880e82369fd70ed96 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f82caa48b6ce457880e82369fd70ed96. Latest consensus state: current_term: 1 leader_uuid: "f82caa48b6ce457880e82369fd70ed96" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f82caa48b6ce457880e82369fd70ed96" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43267 } } }
I20260926 01:12:21.383533 12672 sys_catalog.cc:458] T 00000000000000000000000000000000 P f82caa48b6ce457880e82369fd70ed96 [sys.catalog]: This master's current role is: LEADER
I20260926 01:12:21.383636 12676 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:12:21.383919 12667 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 01:12:21.384192 12676 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:12:21.389066 12676 catalog_manager.cc:1347] Generated new cluster ID: 65570b55d5de4c5d8dac84d52dc7f098
I20260926 01:12:21.389132 12676 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 01:12:21 dist-test-slave-7b22 krb5kdc[12155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385141, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:12:21.396682 12681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:21.383229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47916 (local address 127.0.98.62:43267)
0926 01:12:21.383395 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0926 01:12:21.383405 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 01:12:21.383473 (+ 68us) server_negotiation.cc:407] Connection header received
0926 01:12:21.383646 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:12:21.383654 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:12:21.383715 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:12:21.383847 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:12:21.385611 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:12:21.387929 (+ 2318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:12:21.389039 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:12:21.389679 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:12:21.389764 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:12:21.392790 (+ 3026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:12:21.392794 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:12:21.392795 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:12:21.392809 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:12:21.394831 (+ 2022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:12:21.395242 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:12:21.395245 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:12:21.395246 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:12:21.395368 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:12:21.395573 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:12:21.395574 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:12:21.395575 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:12:21.395904 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:12:21.396029 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0926 01:12:21.396294 (+ 265us) server_negotiation.cc:299] Negotiation successful
0926 01:12:21.396434 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":73,"threads_started":1}
I20260926 01:12:21.397610 12676 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:12:21.398368 12676 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:12:21.401937 12676 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f82caa48b6ce457880e82369fd70ed96: Generated new TSK 0
I20260926 01:12:21.402103 12676 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 01:12:22.089455 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:38041/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43267
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:12:22.188066 12713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:12:22.188268 12713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:12:22.188302 12713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:12:22.189551 12713 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:12:22.189646 12713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:12:22.189700 12713 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 01:12:22.191532 12713 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:38041/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43267
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12713
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:12:22.191793 12713 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:12:22.192009 12713 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:12:22.196926 12719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:12:22.196834 12718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:12:22.196836 12721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:12:22.197453 12713 server_base.cc:1034] running on GCE node
I20260926 01:12:22.197660 12713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:12:22.197943 12713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:12:22.199123 12713 hybrid_clock.cc:648] HybridClock initialized: now 1790385142199102 us; error 34 us; skew 500 ppm
Sep 26 01:12:22 dist-test-slave-7b22 krb5kdc[12155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385142, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:12:22.201783 12713 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 01:12:22.202298 12713 webserver.cc:460] Webserver started at http://127.0.98.1:43037/ using document root <none> and password file <none>
I20260926 01:12:22.202499 12713 fs_manager.cc:359] Metadata directory not provided
I20260926 01:12:22.202544 12713 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:12:22.202666 12713 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 01:12:22 dist-test-slave-7b22 krb5kdc[12155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385142, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 01:12:22.218294 12713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/ts-0/data/instance:
uuid: "0a4be25da75343968fa10e8f6229d8f9"
format_stamp: "Formatted at 2026-09-26 01:12:22 on dist-test-slave-7b22"
server_key: "0c6abeb1403f6d38ea0c159097b628af"
server_key_iv: "180467e278365a98f3fdd6a78b665a74"
server_key_version: "kuduclusterkey@0"
I20260926 01:12:22.219013 12713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/ts-0/wal/instance:
uuid: "0a4be25da75343968fa10e8f6229d8f9"
format_stamp: "Formatted at 2026-09-26 01:12:22 on dist-test-slave-7b22"
server_key: "0c6abeb1403f6d38ea0c159097b628af"
server_key_iv: "180467e278365a98f3fdd6a78b665a74"
server_key_version: "kuduclusterkey@0"
I20260926 01:12:22.220937 12713 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260926 01:12:22.234413 12730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:12:22.234620 12713 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 01:12:22.234714 12713 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/ts-0/wal
uuid: "0a4be25da75343968fa10e8f6229d8f9"
format_stamp: "Formatted at 2026-09-26 01:12:22 on dist-test-slave-7b22"
server_key: "0c6abeb1403f6d38ea0c159097b628af"
server_key_iv: "180467e278365a98f3fdd6a78b665a74"
server_key_version: "kuduclusterkey@0"
I20260926 01:12:22.234815 12713 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:12:22.279196 12713 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:12:22.281150 12713 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:12:22.281283 12713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:12:22.281513 12713 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:12:22.281888 12713 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:12:22.281932 12713 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:12:22.281963 12713 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:12:22.281998 12713 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:12:22.298882 12713 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44545
I20260926 01:12:22.299371 12713 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/ts-0/data/info.pb
I20260926 01:12:22.306993 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 12713
I20260926 01:12:22.307099 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/ts-0/wal/instance
I20260926 01:12:22.307288 392 external_mini_cluster.cc:1428] Setting key 2640949b6a154712c0263fbabd9c0285
I20260926 01:12:22.302744 12843 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44545 every 8 connection(s)
Sep 26 01:12:22 dist-test-slave-7b22 krb5kdc[12155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385142, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:12:22.317358 12846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:22.302960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43267 (local address 127.0.98.1:56001)
0926 01:12:22.303267 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0926 01:12:22.303330 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0926 01:12:22.304063 (+ 733us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:12:22.304578 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:12:22.304583 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:12:22.304837 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:12:22.305156 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:12:22.305162 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:12:22.306219 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:12:22.306221 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:12:22.310476 (+ 4255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:12:22.310480 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:12:22.311065 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:12:22.311066 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:12:22.311214 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:12:22.311852 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:12:22.311863 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:12:22.313592 (+ 1729us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:12:22.315628 (+ 2036us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:12:22.315636 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:12:22.315639 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:12:22.315938 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:12:22.316149 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:12:22.316150 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:12:22.316150 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:12:22.316307 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:12:22.316628 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:12:22.316635 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:12:22.316823 (+ 188us) client_negotiation.cc:769] Sending connection context
0926 01:12:22.316937 (+ 114us) client_negotiation.cc:240] Negotiation successful
0926 01:12:22.317100 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":96,"threads_started":1}
I20260926 01:12:22.318004 12847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:22.303105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56001 (local address 127.0.98.62:43267)
0926 01:12:22.304294 (+ 1189us) server_negotiation.cc:206] Beginning negotiation
0926 01:12:22.304296 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:12:22.304303 (+ 7us) server_negotiation.cc:407] Connection header received
0926 01:12:22.304348 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:12:22.304354 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:12:22.304372 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:12:22.304437 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:12:22.305267 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:12:22.306111 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:12:22.310591 (+ 4480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:12:22.310910 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:12:22.310978 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:12:22.313710 (+ 2732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:12:22.313712 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:12:22.313712 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:12:22.313720 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:12:22.315534 (+ 1814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:12:22.316019 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:12:22.316020 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:12:22.316021 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:12:22.316084 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:12:22.316390 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:12:22.316391 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:12:22.316392 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:12:22.316540 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:12:22.316602 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 01:12:22.317777 (+ 1175us) server_negotiation.cc:299] Negotiation successful
0926 01:12:22.317904 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1153,"thread_start_us":82,"threads_started":1}
I20260926 01:12:22.318588 12844 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43267
I20260926 01:12:22.318692 12844 heartbeater.cc:464] Registering TS with master...
I20260926 01:12:22.318984 12844 heartbeater.cc:511] Master 127.0.98.62:43267 requested a full tablet report, sending...
I20260926 01:12:22.319692 12631 ts_manager.cc:194] Registered new tserver with Master: 0a4be25da75343968fa10e8f6229d8f9 (127.0.98.1:44545)
I20260926 01:12:22.320598 12631 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:56001
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 01:12:22.344975 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:38041/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43267
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 01:12:22 dist-test-slave-7b22 krb5kdc[12155](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790385142, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260926 01:12:22.465801 12852 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:12:22.466131 12852 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:12:22.466166 12852 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:12:22.467917 12852 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:12:22.467968 12852 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:12:22.468006 12852 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 01:12:22.470135 12852 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:38041/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43267
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12852
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:12:22.470396 12852 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:12:22.470691 12852 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:12:22.477090 12860 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:12:22.477881 12852 server_base.cc:1034] running on GCE node
W20260926 01:12:22.478214 12857 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:12:22.478457 12858 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:12:22.478699 12852 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:12:22.478976 12852 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:12:22.480134 12852 hybrid_clock.cc:648] HybridClock initialized: now 1790385142480111 us; error 35 us; skew 500 ppm
Sep 26 01:12:22 dist-test-slave-7b22 krb5kdc[12155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385142, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:12:22.483345 12852 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 01:12:22.483875 12852 webserver.cc:460] Webserver started at http://127.0.98.2:42117/ using document root <none> and password file <none>
I20260926 01:12:22.484126 12852 fs_manager.cc:359] Metadata directory not provided
I20260926 01:12:22.484174 12852 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:12:22.484294 12852 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 01:12:22 dist-test-slave-7b22 krb5kdc[12155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385142, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 01:12:22.498142 12852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/ts-1/data/instance:
uuid: "416d2ffd801f4c62a95304d024686028"
format_stamp: "Formatted at 2026-09-26 01:12:22 on dist-test-slave-7b22"
server_key: "ae50856b60e3f4d8729d829e9e7887e4"
server_key_iv: "70bce6e2069a7d5d55e3c5eef42152c8"
server_key_version: "kuduclusterkey@0"
I20260926 01:12:22.498690 12852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/ts-1/wal/instance:
uuid: "416d2ffd801f4c62a95304d024686028"
format_stamp: "Formatted at 2026-09-26 01:12:22 on dist-test-slave-7b22"
server_key: "ae50856b60e3f4d8729d829e9e7887e4"
server_key_iv: "70bce6e2069a7d5d55e3c5eef42152c8"
server_key_version: "kuduclusterkey@0"
I20260926 01:12:22.509557 12852 fs_manager.cc:691] Time spent creating directory manager: real 0.011s user 0.001s sys 0.000s
I20260926 01:12:22.532691 12868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:12:22.533004 12852 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.001s
I20260926 01:12:22.533115 12852 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/ts-1/wal
uuid: "416d2ffd801f4c62a95304d024686028"
format_stamp: "Formatted at 2026-09-26 01:12:22 on dist-test-slave-7b22"
server_key: "ae50856b60e3f4d8729d829e9e7887e4"
server_key_iv: "70bce6e2069a7d5d55e3c5eef42152c8"
server_key_version: "kuduclusterkey@0"
I20260926 01:12:22.533241 12852 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:12:22.570009 12852 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:12:22.572520 12852 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:12:22.572738 12852 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:12:22.573046 12852 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:12:22.573469 12852 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:12:22.573557 12852 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:12:22.573624 12852 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:12:22.573674 12852 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:12:22.580740 12852 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33425
I20260926 01:12:22.581406 12852 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/ts-1/data/info.pb
I20260926 01:12:22.584422 12981 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33425 every 8 connection(s)
I20260926 01:12:22.586211 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 12852
I20260926 01:12:22.586316 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/ts-1/wal/instance
I20260926 01:12:22.586508 392 external_mini_cluster.cc:1428] Setting key 847aaf414ac9def258b7a8b4b452adce
Sep 26 01:12:22 dist-test-slave-7b22 krb5kdc[12155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385142, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260926 01:12:22.606575 12847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:22.585372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36791 (local address 127.0.98.62:43267)
0926 01:12:22.585423 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 01:12:22.585425 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:12:22.586074 (+ 649us) server_negotiation.cc:407] Connection header received
0926 01:12:22.586897 (+ 823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:12:22.586898 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:12:22.586910 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:12:22.586938 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:12:22.587829 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:12:22.588641 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:12:22.590188 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:12:22.590523 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:12:22.590570 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:12:22.598856 (+ 8286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:12:22.598859 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:12:22.598859 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:12:22.598869 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:12:22.600600 (+ 1731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:12:22.605149 (+ 4549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:12:22.605151 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:12:22.605152 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:12:22.605244 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:12:22.605641 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:12:22.605642 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:12:22.605643 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:12:22.605811 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:12:22.605904 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 01:12:22.606358 (+ 454us) server_negotiation.cc:299] Negotiation successful
0926 01:12:22.606468 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 01:12:22.607432 12984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:22.585064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43267 (local address 127.0.98.2:36791)
0926 01:12:22.585941 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0926 01:12:22.585985 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 01:12:22.586776 (+ 791us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:12:22.587140 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:12:22.587145 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:12:22.587430 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:12:22.587723 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:12:22.587730 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:12:22.588758 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:12:22.588759 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:12:22.590090 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:12:22.590093 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:12:22.590677 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:12:22.590678 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:12:22.590834 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:12:22.596809 (+ 5975us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:12:22.596823 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:12:22.598694 (+ 1871us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:12:22.600718 (+ 2024us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:12:22.600731 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:12:22.600735 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:12:22.601091 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:12:22.605342 (+ 4251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:12:22.605343 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:12:22.605344 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:12:22.605557 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:12:22.605943 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:12:22.605953 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:12:22.606183 (+ 230us) client_negotiation.cc:769] Sending connection context
0926 01:12:22.606977 (+ 794us) client_negotiation.cc:240] Negotiation successful
0926 01:12:22.607179 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":834,"thread_start_us":100,"threads_started":1}
I20260926 01:12:22.608278 12982 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43267
I20260926 01:12:22.608376 12982 heartbeater.cc:464] Registering TS with master...
I20260926 01:12:22.608563 12982 heartbeater.cc:511] Master 127.0.98.62:43267 requested a full tablet report, sending...
I20260926 01:12:22.608965 12631 ts_manager.cc:194] Registered new tserver with Master: 416d2ffd801f4c62a95304d024686028 (127.0.98.2:33425)
I20260926 01:12:22.609468 12631 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:36791
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 01:12:22.632854 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:38041/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43267
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:12:22.738420 12987 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:12:22.738675 12987 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:12:22.738710 12987 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:12:22.740092 12987 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:12:22.740144 12987 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:12:22.740180 12987 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 01:12:22.742385 12987 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:38041/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43267
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12987
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:12:22.742805 12987 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:12:22.743103 12987 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:12:22.750093 12992 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:12:22.750109 12993 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:12:22.750396 12995 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:12:22.750591 12987 server_base.cc:1034] running on GCE node
I20260926 01:12:22.750790 12987 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:12:22.751039 12987 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:12:22.752207 12987 hybrid_clock.cc:648] HybridClock initialized: now 1790385142752179 us; error 46 us; skew 500 ppm
Sep 26 01:12:22 dist-test-slave-7b22 krb5kdc[12155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385142, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:12:22.755198 12987 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 01:12:22.755693 12987 webserver.cc:460] Webserver started at http://127.0.98.3:44569/ using document root <none> and password file <none>
I20260926 01:12:22.755911 12987 fs_manager.cc:359] Metadata directory not provided
I20260926 01:12:22.755962 12987 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:12:22.756121 12987 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 01:12:22 dist-test-slave-7b22 krb5kdc[12155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385142, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 01:12:22.770001 12987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/ts-2/data/instance:
uuid: "66b49e280a66492b9f4d6678238efe82"
format_stamp: "Formatted at 2026-09-26 01:12:22 on dist-test-slave-7b22"
server_key: "48f5ea7e16acc6e28f6dcad435b32902"
server_key_iv: "d73e37d6dd307216f08e310e2e199577"
server_key_version: "kuduclusterkey@0"
I20260926 01:12:22.770533 12987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/ts-2/wal/instance:
uuid: "66b49e280a66492b9f4d6678238efe82"
format_stamp: "Formatted at 2026-09-26 01:12:22 on dist-test-slave-7b22"
server_key: "48f5ea7e16acc6e28f6dcad435b32902"
server_key_iv: "d73e37d6dd307216f08e310e2e199577"
server_key_version: "kuduclusterkey@0"
I20260926 01:12:22.774219 12987 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260926 01:12:22.798029 13002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:12:22.798317 12987 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.001s sys 0.000s
I20260926 01:12:22.798425 12987 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/ts-2/wal
uuid: "66b49e280a66492b9f4d6678238efe82"
format_stamp: "Formatted at 2026-09-26 01:12:22 on dist-test-slave-7b22"
server_key: "48f5ea7e16acc6e28f6dcad435b32902"
server_key_iv: "d73e37d6dd307216f08e310e2e199577"
server_key_version: "kuduclusterkey@0"
I20260926 01:12:22.798529 12987 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:12:22.814986 12987 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:12:22.817600 12987 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:12:22.817865 12987 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:12:22.818116 12987 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:12:22.818526 12987 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:12:22.818609 12987 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:12:22.818668 12987 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:12:22.818717 12987 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:12:22.825027 12987 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40123
I20260926 01:12:22.825558 12987 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/ts-2/data/info.pb
I20260926 01:12:22.832192 13115 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40123 every 8 connection(s)
I20260926 01:12:22.833432 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 12987
I20260926 01:12:22.833541 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/ts-2/wal/instance
I20260926 01:12:22.833721 392 external_mini_cluster.cc:1428] Setting key 62dfc0543c86ecc8a547e0fe1f990328
Sep 26 01:12:22 dist-test-slave-7b22 krb5kdc[12155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385142, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:12:22.844842 12847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:22.832814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33259 (local address 127.0.98.62:43267)
0926 01:12:22.832915 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0926 01:12:22.832918 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:12:22.833061 (+ 143us) server_negotiation.cc:407] Connection header received
0926 01:12:22.833737 (+ 676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:12:22.833740 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:12:22.833753 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:12:22.833855 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:12:22.835435 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:12:22.836142 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:12:22.837841 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:12:22.838203 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:12:22.838308 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:12:22.841316 (+ 3008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:12:22.841318 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:12:22.841318 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:12:22.841328 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:12:22.843065 (+ 1737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:12:22.843601 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:12:22.843603 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:12:22.843604 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:12:22.843667 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:12:22.843974 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:12:22.843975 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:12:22.843976 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:12:22.844139 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:12:22.844209 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 01:12:22.844645 (+ 436us) server_negotiation.cc:299] Negotiation successful
0926 01:12:22.844739 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 01:12:22.845285 13118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:22.832463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43267 (local address 127.0.98.3:33259)
0926 01:12:22.832919 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0926 01:12:22.832966 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 01:12:22.833610 (+ 644us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:12:22.834706 (+ 1096us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:12:22.834712 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:12:22.834996 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:12:22.835294 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:12:22.835302 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:12:22.836263 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:12:22.836264 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:12:22.837679 (+ 1415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:12:22.837682 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:12:22.838382 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:12:22.838383 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:12:22.838592 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:12:22.839191 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:12:22.839210 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:12:22.841182 (+ 1972us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:12:22.843144 (+ 1962us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:12:22.843153 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:12:22.843156 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:12:22.843502 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:12:22.843734 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:12:22.843734 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:12:22.843735 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:12:22.843883 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:12:22.844227 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:12:22.844234 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:12:22.844462 (+ 228us) client_negotiation.cc:769] Sending connection context
0926 01:12:22.844581 (+ 119us) client_negotiation.cc:240] Negotiation successful
0926 01:12:22.844752 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":118,"threads_started":1}
I20260926 01:12:22.845935 13116 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43267
I20260926 01:12:22.846057 13116 heartbeater.cc:464] Registering TS with master...
I20260926 01:12:22.846261 13116 heartbeater.cc:511] Master 127.0.98.62:43267 requested a full tablet report, sending...
I20260926 01:12:22.846698 12631 ts_manager.cc:194] Registered new tserver with Master: 66b49e280a66492b9f4d6678238efe82 (127.0.98.3:40123)
I20260926 01:12:22.847218 12631 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:33259
I20260926 01:12:22.856736 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 01:12:22.857479 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 12713
I20260926 01:12:22.864329 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 12852
I20260926 01:12:22.869776 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 12987
I20260926 01:12:22.874254 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 12599
I20260926 01:12:22.879217 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43267
--webserver_interface=127.0.98.62
--webserver_port=45621
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43267
--ranger_config_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:38041/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:38041/kms with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:12:22.960356 13119 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:12:22.960570 13119 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:12:22.960590 13119 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:12:22.961699 13119 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:12:22.961737 13119 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:12:22.961750 13119 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:12:22.961760 13119 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:12:22.961771 13119 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:12:22.963387 13119 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:38041/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43267
--ranger_config_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43267
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=45621
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13119
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:12:22.963585 13119 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:12:22.963789 13119 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:12:22.968537 13127 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:12:22.968537 13124 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:12:22.968612 13125 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:12:22.968703 13119 server_base.cc:1034] running on GCE node
I20260926 01:12:22.969053 13119 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:12:22.969312 13119 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:12:22.970450 13119 hybrid_clock.cc:648] HybridClock initialized: now 1790385142970437 us; error 32 us; skew 500 ppm
Sep 26 01:12:22 dist-test-slave-7b22 krb5kdc[12155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385142, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:12:22.972796 13119 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 01:12:22.973275 13119 webserver.cc:460] Webserver started at http://127.0.98.62:45621/ using document root <none> and password file <none>
I20260926 01:12:22.973481 13119 fs_manager.cc:359] Metadata directory not provided
I20260926 01:12:22.973536 13119 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 01:12:23 dist-test-slave-7b22 krb5kdc[12155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385142, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 01:12:23.031782 13119 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:12:23.032547 13135 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:12:23.032724 13119 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:12:23.032826 13119 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/master-0/wal
uuid: "f82caa48b6ce457880e82369fd70ed96"
format_stamp: "Formatted at 2026-09-26 01:12:21 on dist-test-slave-7b22"
server_key: "d44b7968ead54ca8d4b4b02b93d1f594"
server_key_iv: "f9d511ec6a8d6186e39a9da822b683a3"
server_key_version: "kuduclusterkey@0"
I20260926 01:12:23.033310 13119 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:12:23.052428 13119 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:12:23.054163 13119 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:12:23.054276 13119 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:12:23.058347 13119 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43267
I20260926 01:12:23.058390 13187 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43267 every 8 connection(s)
I20260926 01:12:23.058737 13119 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
I20260926 01:12:23.059952 13188 sys_catalog.cc:263] Verifying existing consensus state
I20260926 01:12:23.060598 13188 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f82caa48b6ce457880e82369fd70ed96: Bootstrap starting.
I20260926 01:12:23.065301 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 13119
I20260926 01:12:23.065565 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:38041/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:44545
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:43267
--webserver_port=43037
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:38041/kms with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 01:12:23.068259 13188 log.cc:824] T 00000000000000000000000000000000 P f82caa48b6ce457880e82369fd70ed96: Log is configured to *not* fsync() on all Append() calls
I20260926 01:12:23.070082 13188 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f82caa48b6ce457880e82369fd70ed96: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 01:12:23.070351 13188 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f82caa48b6ce457880e82369fd70ed96: Bootstrap complete.
I20260926 01:12:23.072594 13188 raft_consensus.cc:348] T 00000000000000000000000000000000 P f82caa48b6ce457880e82369fd70ed96 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f82caa48b6ce457880e82369fd70ed96" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43267 } }
I20260926 01:12:23.072844 13188 raft_consensus.cc:729] T 00000000000000000000000000000000 P f82caa48b6ce457880e82369fd70ed96 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f82caa48b6ce457880e82369fd70ed96, State: Initialized, Role: FOLLOWER
I20260926 01:12:23.073016 13188 consensus_queue.cc:260] T 00000000000000000000000000000000 P f82caa48b6ce457880e82369fd70ed96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f82caa48b6ce457880e82369fd70ed96" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43267 } }
I20260926 01:12:23.073089 13188 raft_consensus.cc:388] T 00000000000000000000000000000000 P f82caa48b6ce457880e82369fd70ed96 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:12:23.073139 13188 raft_consensus.cc:482] T 00000000000000000000000000000000 P f82caa48b6ce457880e82369fd70ed96 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:12:23.073184 13188 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f82caa48b6ce457880e82369fd70ed96 [term 1 FOLLOWER]: Advancing to term 2
I20260926 01:12:23.074267 13188 raft_consensus.cc:504] T 00000000000000000000000000000000 P f82caa48b6ce457880e82369fd70ed96 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f82caa48b6ce457880e82369fd70ed96" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43267 } }
I20260926 01:12:23.074368 13188 leader_election.cc:302] T 00000000000000000000000000000000 P f82caa48b6ce457880e82369fd70ed96 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f82caa48b6ce457880e82369fd70ed96; no voters:
I20260926 01:12:23.074541 13188 leader_election.cc:288] T 00000000000000000000000000000000 P f82caa48b6ce457880e82369fd70ed96 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 01:12:23.074611 13192 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f82caa48b6ce457880e82369fd70ed96 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 01:12:23.074805 13188 sys_catalog.cc:564] T 00000000000000000000000000000000 P f82caa48b6ce457880e82369fd70ed96 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:12:23.074813 13192 raft_consensus.cc:686] T 00000000000000000000000000000000 P f82caa48b6ce457880e82369fd70ed96 [term 2 LEADER]: Becoming Leader. State: Replica: f82caa48b6ce457880e82369fd70ed96, State: Running, Role: LEADER
I20260926 01:12:23.074954 13192 consensus_queue.cc:237] T 00000000000000000000000000000000 P f82caa48b6ce457880e82369fd70ed96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f82caa48b6ce457880e82369fd70ed96" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43267 } }
I20260926 01:12:23.075309 13193 sys_catalog.cc:455] T 00000000000000000000000000000000 P f82caa48b6ce457880e82369fd70ed96 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "f82caa48b6ce457880e82369fd70ed96" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f82caa48b6ce457880e82369fd70ed96" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43267 } } }
I20260926 01:12:23.075314 13194 sys_catalog.cc:455] T 00000000000000000000000000000000 P f82caa48b6ce457880e82369fd70ed96 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f82caa48b6ce457880e82369fd70ed96. Latest consensus state: current_term: 2 leader_uuid: "f82caa48b6ce457880e82369fd70ed96" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f82caa48b6ce457880e82369fd70ed96" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43267 } } }
I20260926 01:12:23.075512 13194 sys_catalog.cc:458] T 00000000000000000000000000000000 P f82caa48b6ce457880e82369fd70ed96 [sys.catalog]: This master's current role is: LEADER
I20260926 01:12:23.075699 13193 sys_catalog.cc:458] T 00000000000000000000000000000000 P f82caa48b6ce457880e82369fd70ed96 [sys.catalog]: This master's current role is: LEADER
I20260926 01:12:23.075695 13201 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:12:23.076277 13201 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:12:23.076400 13201 catalog_manager.cc:1259] Loaded cluster ID: 65570b55d5de4c5d8dac84d52dc7f098
I20260926 01:12:23.076476 13201 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 01:12:23.076687 13188 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 01:12:23.077725 13201 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:12:23.077936 13201 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f82caa48b6ce457880e82369fd70ed96: Loaded TSK: 0
I20260926 01:12:23.078161 13201 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 01:12:23.150475 13190 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:12:23.150672 13190 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:12:23.150704 13190 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:12:23.151887 13190 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:12:23.151928 13190 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:12:23.151962 13190 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 01:12:23.153411 13190 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:38041/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:44545
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=43037
--tserver_master_addrs=127.0.98.62:43267
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13190
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:12:23.153648 13190 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:12:23.153898 13190 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:12:23.160238 13226 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:12:23.160468 13224 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:12:23.161062 13223 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:12:23.162235 13190 server_base.cc:1034] running on GCE node
I20260926 01:12:23.162479 13190 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:12:23.162784 13190 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:12:23.163957 13190 hybrid_clock.cc:648] HybridClock initialized: now 1790385143163929 us; error 43 us; skew 500 ppm
Sep 26 01:12:23 dist-test-slave-7b22 krb5kdc[12155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385143, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:12:23.167379 13190 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 01:12:23.167987 13190 webserver.cc:460] Webserver started at http://127.0.98.1:43037/ using document root <none> and password file <none>
I20260926 01:12:23.168246 13190 fs_manager.cc:359] Metadata directory not provided
I20260926 01:12:23.168313 13190 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 01:12:23 dist-test-slave-7b22 krb5kdc[12155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385143, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 01:12:23.181910 13190 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260926 01:12:23.182664 13236 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:12:23.182806 13190 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 01:12:23.182880 13190 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/ts-0/wal
uuid: "0a4be25da75343968fa10e8f6229d8f9"
format_stamp: "Formatted at 2026-09-26 01:12:22 on dist-test-slave-7b22"
server_key: "0c6abeb1403f6d38ea0c159097b628af"
server_key_iv: "180467e278365a98f3fdd6a78b665a74"
server_key_version: "kuduclusterkey@0"
I20260926 01:12:23.183316 13190 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:12:23.193876 13190 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:12:23.195832 13190 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:12:23.195971 13190 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:12:23.196170 13190 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:12:23.196499 13190 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:12:23.196535 13190 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:12:23.196565 13190 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:12:23.196602 13190 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:12:23.203099 13190 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44545
I20260926 01:12:23.203485 13190 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/ts-0/data/info.pb
I20260926 01:12:23.203951 13349 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44545 every 8 connection(s)
I20260926 01:12:23.211375 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 13190
I20260926 01:12:23.211651 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:38041/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:33425
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:43267
--webserver_port=42117
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:38041/kms with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 01:12:23 dist-test-slave-7b22 krb5kdc[12155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385143, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:12:23.222129 13352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:23.204205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43267 (local address 127.0.98.1:51557)
0926 01:12:23.204348 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 01:12:23.204388 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 01:12:23.204841 (+ 453us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:12:23.210456 (+ 5615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:12:23.210460 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:12:23.210708 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:12:23.210921 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:12:23.210927 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:12:23.213198 (+ 2271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:12:23.213200 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:12:23.214433 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:12:23.214437 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:12:23.215325 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:12:23.215327 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:12:23.215454 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:12:23.216075 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:12:23.216084 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:12:23.217732 (+ 1648us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:12:23.220013 (+ 2281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:12:23.220023 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:12:23.220026 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:12:23.220339 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:12:23.220712 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:12:23.220714 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:12:23.220714 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:12:23.220852 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:12:23.221363 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:12:23.221369 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:12:23.221539 (+ 170us) client_negotiation.cc:769] Sending connection context
0926 01:12:23.221652 (+ 113us) client_negotiation.cc:240] Negotiation successful
0926 01:12:23.221868 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":77,"threads_started":1}
I20260926 01:12:23.222957 13353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:23.204283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51557 (local address 127.0.98.62:43267)
0926 01:12:23.209990 (+ 5707us) server_negotiation.cc:206] Beginning negotiation
0926 01:12:23.210001 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 01:12:23.210042 (+ 41us) server_negotiation.cc:407] Connection header received
0926 01:12:23.210191 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:12:23.210197 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:12:23.210246 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:12:23.210390 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:12:23.211055 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:12:23.213068 (+ 2013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:12:23.214526 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:12:23.215118 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:12:23.215218 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:12:23.217912 (+ 2694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:12:23.217917 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:12:23.217918 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:12:23.217932 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:12:23.219894 (+ 1962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:12:23.220440 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:12:23.220494 (+ 54us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:12:23.220495 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:12:23.220633 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:12:23.220924 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:12:23.220925 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:12:23.220925 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:12:23.221257 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:12:23.221333 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 01:12:23.222581 (+ 1248us) server_negotiation.cc:299] Negotiation successful
0926 01:12:23.222708 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5641,"thread_start_us":86,"threads_started":1}
I20260926 01:12:23.224277 13350 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43267
I20260926 01:12:23.224377 13350 heartbeater.cc:464] Registering TS with master...
I20260926 01:12:23.224560 13350 heartbeater.cc:511] Master 127.0.98.62:43267 requested a full tablet report, sending...
W20260926 01:12:23.225369 13350 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:43267 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 01:12:23.225433 13350 heartbeater.cc:464] Registering TS with master...
I20260926 01:12:23.225489 13350 heartbeater.cc:511] Master 127.0.98.62:43267 requested a full tablet report, sending...
I20260926 01:12:23.225888 13350 heartbeater.cc:464] Registering TS with master...
I20260926 01:12:23.225957 13350 heartbeater.cc:511] Master 127.0.98.62:43267 requested a full tablet report, sending...
W20260926 01:12:23.226286 13350 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 01:12:23.312019 13354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:12:23.312377 13354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:12:23.312433 13354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:12:23.314227 13354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:12:23.314316 13354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:12:23.314369 13354 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 01:12:23.316614 13354 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:38041/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:33425
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=42117
--tserver_master_addrs=127.0.98.62:43267
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13354
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:12:23.317000 13354 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:12:23.317307 13354 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:12:23.323004 13360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:12:23.323544 13354 server_base.cc:1034] running on GCE node
W20260926 01:12:23.325966 13362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:12:23.328244 13359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:12:23.328560 13354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:12:23.328833 13354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:12:23.330008 13354 hybrid_clock.cc:648] HybridClock initialized: now 1790385143329996 us; error 30 us; skew 500 ppm
Sep 26 01:12:23 dist-test-slave-7b22 krb5kdc[12155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385143, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:12:23.332629 13354 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 01:12:23.333117 13354 webserver.cc:460] Webserver started at http://127.0.98.2:42117/ using document root <none> and password file <none>
I20260926 01:12:23.333341 13354 fs_manager.cc:359] Metadata directory not provided
I20260926 01:12:23.333401 13354 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 01:12:23 dist-test-slave-7b22 krb5kdc[12155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385143, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 01:12:23.362499 13354 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.001s sys 0.000s
I20260926 01:12:23.363222 13370 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:12:23.363394 13354 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 01:12:23.363484 13354 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/ts-1/wal
uuid: "416d2ffd801f4c62a95304d024686028"
format_stamp: "Formatted at 2026-09-26 01:12:22 on dist-test-slave-7b22"
server_key: "ae50856b60e3f4d8729d829e9e7887e4"
server_key_iv: "70bce6e2069a7d5d55e3c5eef42152c8"
server_key_version: "kuduclusterkey@0"
I20260926 01:12:23.363837 13354 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:12:23.386648 13354 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:12:23.388590 13354 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:12:23.388734 13354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:12:23.388957 13354 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:12:23.389266 13354 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:12:23.389339 13354 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:12:23.389380 13354 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:12:23.389398 13354 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:12:23.395699 13354 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33425
I20260926 01:12:23.396098 13354 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/ts-1/data/info.pb
I20260926 01:12:23.396533 13483 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33425 every 8 connection(s)
I20260926 01:12:23.399571 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 13354
I20260926 01:12:23.399837 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:38041/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:40123
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:43267
--webserver_port=44569
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:38041/kms with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 01:12:23 dist-test-slave-7b22 krb5kdc[12155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385143, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:12:23.417289 13353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:23.396875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55337 (local address 127.0.98.62:43267)
0926 01:12:23.396920 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 01:12:23.396922 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:12:23.397439 (+ 517us) server_negotiation.cc:407] Connection header received
0926 01:12:23.398158 (+ 719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:12:23.398159 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:12:23.398177 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:12:23.398265 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:12:23.399030 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:12:23.399807 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:12:23.401311 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:12:23.401629 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:12:23.401679 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:12:23.414383 (+ 12704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:12:23.414385 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:12:23.414385 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:12:23.414393 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:12:23.415838 (+ 1445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:12:23.416295 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:12:23.416296 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:12:23.416297 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:12:23.416350 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:12:23.416598 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:12:23.416599 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:12:23.416600 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:12:23.416721 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:12:23.416776 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 01:12:23.417092 (+ 316us) server_negotiation.cc:299] Negotiation successful
0926 01:12:23.417186 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 01:12:23.417469 13486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:23.396921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43267 (local address 127.0.98.2:55337)
0926 01:12:23.397286 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0926 01:12:23.397334 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 01:12:23.398006 (+ 672us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:12:23.398354 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:12:23.398358 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:12:23.398622 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:12:23.398909 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:12:23.398915 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:12:23.399927 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:12:23.399928 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:12:23.401199 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:12:23.401202 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:12:23.401796 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:12:23.401797 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:12:23.401985 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:12:23.402589 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:12:23.402600 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:12:23.414217 (+ 11617us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:12:23.415934 (+ 1717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:12:23.415947 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:12:23.415950 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:12:23.416212 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:12:23.416410 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:12:23.416411 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:12:23.416411 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:12:23.416538 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:12:23.416785 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:12:23.416795 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:12:23.416967 (+ 172us) client_negotiation.cc:769] Sending connection context
0926 01:12:23.417060 (+ 93us) client_negotiation.cc:240] Negotiation successful
0926 01:12:23.417223 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":88,"threads_started":1}
I20260926 01:12:23.418460 13484 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43267
I20260926 01:12:23.418599 13484 heartbeater.cc:464] Registering TS with master...
I20260926 01:12:23.418833 13484 heartbeater.cc:511] Master 127.0.98.62:43267 requested a full tablet report, sending...
W20260926 01:12:23.419579 13484 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:43267 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 01:12:23.419661 13484 heartbeater.cc:464] Registering TS with master...
I20260926 01:12:23.419730 13484 heartbeater.cc:511] Master 127.0.98.62:43267 requested a full tablet report, sending...
I20260926 01:12:23.420104 13484 heartbeater.cc:464] Registering TS with master...
I20260926 01:12:23.420177 13484 heartbeater.cc:511] Master 127.0.98.62:43267 requested a full tablet report, sending...
W20260926 01:12:23.420487 13484 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 01:12:23.492957 13487 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:12:23.493172 13487 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:12:23.493196 13487 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:12:23.494630 13487 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:12:23.494680 13487 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:12:23.494717 13487 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 01:12:23.496271 13487 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:38041/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:40123
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=44569
--tserver_master_addrs=127.0.98.62:43267
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13487
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:12:23.496520 13487 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:12:23.496745 13487 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:12:23.504433 13493 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:12:23.504446 13495 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:12:23.504806 13492 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:12:23.505898 13487 server_base.cc:1034] running on GCE node
I20260926 01:12:23.506127 13487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:12:23.506426 13487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:12:23.507607 13487 hybrid_clock.cc:648] HybridClock initialized: now 1790385143507580 us; error 38 us; skew 500 ppm
Sep 26 01:12:23 dist-test-slave-7b22 krb5kdc[12155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385143, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:12:23.510978 13487 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 01:12:23.511650 13487 webserver.cc:460] Webserver started at http://127.0.98.3:44569/ using document root <none> and password file <none>
I20260926 01:12:23.511919 13487 fs_manager.cc:359] Metadata directory not provided
I20260926 01:12:23.512017 13487 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 01:12:23 dist-test-slave-7b22 krb5kdc[12155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385143, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 01:12:23.538273 13487 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.001s sys 0.000s
I20260926 01:12:23.541097 13503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:12:23.541311 13487 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.000s
I20260926 01:12:23.541412 13487 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/ts-2/wal
uuid: "66b49e280a66492b9f4d6678238efe82"
format_stamp: "Formatted at 2026-09-26 01:12:22 on dist-test-slave-7b22"
server_key: "48f5ea7e16acc6e28f6dcad435b32902"
server_key_iv: "d73e37d6dd307216f08e310e2e199577"
server_key_version: "kuduclusterkey@0"
I20260926 01:12:23.542021 13487 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:12:23.559039 13487 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:12:23.560985 13487 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:12:23.561106 13487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:12:23.561350 13487 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:12:23.561709 13487 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:12:23.561746 13487 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:12:23.561769 13487 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:12:23.561793 13487 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:12:23.568289 13487 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40123
I20260926 01:12:23.569012 13487 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/ts-2/data/info.pb
I20260926 01:12:23.569649 13616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40123 every 8 connection(s)
Sep 26 01:12:23 dist-test-slave-7b22 krb5kdc[12155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385143, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:12:23.577118 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 13487
I20260926 01:12:23.578085 13353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:23.570097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45367 (local address 127.0.98.62:43267)
0926 01:12:23.570137 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 01:12:23.570139 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:12:23.570150 (+ 11us) server_negotiation.cc:407] Connection header received
0926 01:12:23.570577 (+ 427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:12:23.570579 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:12:23.570592 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:12:23.570642 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:12:23.571198 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:12:23.571693 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:12:23.572660 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:12:23.572864 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:12:23.572919 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:12:23.575060 (+ 2141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:12:23.575061 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:12:23.575062 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:12:23.575069 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:12:23.576572 (+ 1503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:12:23.576930 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:12:23.576931 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:12:23.576937 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:12:23.577004 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:12:23.577287 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:12:23.577288 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:12:23.577288 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:12:23.577426 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:12:23.577467 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0926 01:12:23.577890 (+ 423us) server_negotiation.cc:299] Negotiation successful
0926 01:12:23.577993 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 01:12:23.578606 13619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:23.569769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43267 (local address 127.0.98.3:45367)
0926 01:12:23.569936 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 01:12:23.569977 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 01:12:23.570479 (+ 502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:12:23.570711 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:12:23.570715 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:12:23.570898 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:12:23.571091 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:12:23.571095 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:12:23.571778 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:12:23.571779 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:12:23.572579 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:12:23.572582 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:12:23.572942 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:12:23.572943 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:12:23.573060 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:12:23.573528 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:12:23.573536 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:12:23.574974 (+ 1438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:12:23.576647 (+ 1673us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:12:23.576653 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:12:23.576656 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:12:23.576865 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:12:23.577076 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:12:23.577077 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:12:23.577077 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:12:23.577212 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:12:23.577513 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:12:23.577519 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:12:23.577705 (+ 186us) client_negotiation.cc:769] Sending connection context
0926 01:12:23.578209 (+ 504us) client_negotiation.cc:240] Negotiation successful
0926 01:12:23.578377 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":92,"threads_started":1}
I20260926 01:12:23.579034 13617 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43267
I20260926 01:12:23.579131 13617 heartbeater.cc:464] Registering TS with master...
I20260926 01:12:23.579322 13617 heartbeater.cc:511] Master 127.0.98.62:43267 requested a full tablet report, sending...
W20260926 01:12:23.579865 13617 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:43267 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 01:12:23.579933 13617 heartbeater.cc:464] Registering TS with master...
I20260926 01:12:23.580008 13617 heartbeater.cc:511] Master 127.0.98.62:43267 requested a full tablet report, sending...
I20260926 01:12:23.580266 13617 heartbeater.cc:464] Registering TS with master...
I20260926 01:12:23.580336 13617 heartbeater.cc:511] Master 127.0.98.62:43267 requested a full tablet report, sending...
W20260926 01:12:23.580587 13617 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 01:12:23.584957 13621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:23.577527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47920 (local address 127.0.98.62:43267)
0926 01:12:23.577802 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0926 01:12:23.577803 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 01:12:23.577810 (+ 7us) server_negotiation.cc:407] Connection header received
0926 01:12:23.577867 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:12:23.577867 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:12:23.577878 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:12:23.577932 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:12:23.579247 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:12:23.580103 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:12:23.581144 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:12:23.581348 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:12:23.581401 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:12:23.582588 (+ 1187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:12:23.582589 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:12:23.582589 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:12:23.582596 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:12:23.583925 (+ 1329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:12:23.584232 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:12:23.584233 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:12:23.584234 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:12:23.584289 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:12:23.584446 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:12:23.584447 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:12:23.584447 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:12:23.584558 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:12:23.584735 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0926 01:12:23.584769 (+ 34us) server_negotiation.cc:299] Negotiation successful
0926 01:12:23.584841 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":52,"threads_started":1}
Sep 26 01:12:24 dist-test-slave-7b22 krb5kdc[12155](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790385144, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:12:24.226485 13350 heartbeater.cc:464] Registering TS with master...
I20260926 01:12:24.226650 13350 heartbeater.cc:511] Master 127.0.98.62:43267 requested a full tablet report, sending...
I20260926 01:12:24.227100 13152 ts_manager.cc:194] Registered new tserver with Master: 0a4be25da75343968fa10e8f6229d8f9 (127.0.98.1:44545)
I20260926 01:12:24.227939 13152 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:51557
I20260926 01:12:24.420699 13484 heartbeater.cc:464] Registering TS with master...
I20260926 01:12:24.420871 13484 heartbeater.cc:511] Master 127.0.98.62:43267 requested a full tablet report, sending...
I20260926 01:12:24.421428 13152 ts_manager.cc:194] Registered new tserver with Master: 416d2ffd801f4c62a95304d024686028 (127.0.98.2:33425)
I20260926 01:12:24.422003 13152 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:55337
Sep 26 01:12:24 dist-test-slave-7b22 krb5kdc[12155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385144, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 01:12:24.580793 13617 heartbeater.cc:464] Registering TS with master...
I20260926 01:12:24.580957 13617 heartbeater.cc:511] Master 127.0.98.62:43267 requested a full tablet report, sending...
I20260926 01:12:24.581298 13152 ts_manager.cc:194] Registered new tserver with Master: 66b49e280a66492b9f4d6678238efe82 (127.0.98.3:40123)
I20260926 01:12:24.581691 13152 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:45367
I20260926 01:12:24.589540 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 01:12:24.596439 13640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:24.590558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47928 (local address 127.0.98.62:43267)
0926 01:12:24.590671 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0926 01:12:24.590673 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:12:24.590698 (+ 25us) server_negotiation.cc:407] Connection header received
0926 01:12:24.590769 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:12:24.590770 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:12:24.590781 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:12:24.590849 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:12:24.591361 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:12:24.591875 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:12:24.592594 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:12:24.592808 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:12:24.592870 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:12:24.594086 (+ 1216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:12:24.594088 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:12:24.594089 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:12:24.594096 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:12:24.595423 (+ 1327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:12:24.595729 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:12:24.595731 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:12:24.595731 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:12:24.595784 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:12:24.595947 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:12:24.595948 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:12:24.595948 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:12:24.596057 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:12:24.596144 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 01:12:24.596248 (+ 104us) server_negotiation.cc:299] Negotiation successful
0926 01:12:24.596333 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":64,"threads_started":1}
I20260926 01:12:24.598368 13152 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47928:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:12:24.600042 13152 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:12:24.606769 13647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:24.602961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40123 (local address 127.0.0.1:57130)
0926 01:12:24.603084 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0926 01:12:24.603092 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 01:12:24.603131 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:12:24.603686 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:12:24.603689 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:12:24.603691 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:12:24.603766 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:12:24.603768 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:12:24.604600 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:12:24.604601 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:12:24.605584 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:12:24.605587 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:12:24.606487 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:12:24.606488 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:12:24.606569 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:12:24.606584 (+ 15us) client_negotiation.cc:769] Sending connection context
0926 01:12:24.606630 (+ 46us) client_negotiation.cc:240] Negotiation successful
0926 01:12:24.606679 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":69,"threads_started":1}
I20260926 01:12:24.607223 13650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:24.603310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57130 (local address 127.0.98.3:40123)
0926 01:12:24.603526 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0926 01:12:24.603529 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:12:24.603539 (+ 10us) server_negotiation.cc:407] Connection header received
0926 01:12:24.603572 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:12:24.603574 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:12:24.603590 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:12:24.603659 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:12:24.603838 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:12:24.604523 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:12:24.605658 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:12:24.606403 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:12:24.606981 (+ 578us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:12:24.607079 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 01:12:24.607107 (+ 28us) server_negotiation.cc:299] Negotiation successful
0926 01:12:24.607148 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":61,"threads_started":1}
I20260926 01:12:24.608000 13551 tablet_service.cc:1467] Processing CreateTablet for tablet 7615406f98fc4059b858e1fa9ab72efe (DEFAULT_TABLE table=test-table [id=e9b6a3205b794dda888e91513a04329b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:12:24.608392 13551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7615406f98fc4059b858e1fa9ab72efe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:12:24.609898 13648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:24.603115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33425 (local address 127.0.0.1:52250)
0926 01:12:24.605937 (+ 2822us) negotiation.cc:107] Waiting for socket to connect
0926 01:12:24.605948 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 01:12:24.606029 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:12:24.606269 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:12:24.606270 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:12:24.606272 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:12:24.606312 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:12:24.606314 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:12:24.607696 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:12:24.607697 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:12:24.608482 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:12:24.608485 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:12:24.609569 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:12:24.609570 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:12:24.609639 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:12:24.609643 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 01:12:24.609697 (+ 54us) client_negotiation.cc:240] Negotiation successful
0926 01:12:24.609736 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2805,"thread_start_us":58,"threads_started":1}
I20260926 01:12:24.610589 13649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:24.603218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52250 (local address 127.0.98.2:33425)
0926 01:12:24.603454 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0926 01:12:24.603460 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 01:12:24.606002 (+ 2542us) server_negotiation.cc:407] Connection header received
0926 01:12:24.606105 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:12:24.606108 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:12:24.606124 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:12:24.606174 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:12:24.606381 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:12:24.607608 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:12:24.608558 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:12:24.609258 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:12:24.609424 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:12:24.609528 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 01:12:24.610169 (+ 641us) server_negotiation.cc:299] Negotiation successful
0926 01:12:24.610216 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":76,"threads_started":1}
I20260926 01:12:24.611920 13645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:24.602683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44545 (local address 127.0.0.1:44214)
0926 01:12:24.602868 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 01:12:24.602909 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 01:12:24.602968 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:12:24.608783 (+ 5815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:12:24.608784 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:12:24.608786 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:12:24.608835 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:12:24.608836 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:12:24.610131 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:12:24.610132 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:12:24.611057 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:12:24.611060 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:12:24.611692 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:12:24.611693 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:12:24.611751 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:12:24.611755 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 01:12:24.611814 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 01:12:24.611853 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":103,"threads_started":1}
I20260926 01:12:24.611936 13646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:24.602749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44214 (local address 127.0.98.1:44545)
0926 01:12:24.608624 (+ 5875us) server_negotiation.cc:206] Beginning negotiation
0926 01:12:24.608628 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 01:12:24.608638 (+ 10us) server_negotiation.cc:407] Connection header received
0926 01:12:24.608670 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:12:24.608672 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:12:24.608688 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:12:24.608899 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:12:24.608974 (+ 75us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:12:24.610045 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:12:24.611136 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:12:24.611613 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:12:24.611673 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:12:24.611740 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 01:12:24.611828 (+ 88us) server_negotiation.cc:299] Negotiation successful
0926 01:12:24.611866 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5837,"thread_start_us":61,"threads_started":1}
I20260926 01:12:24.612687 13418 tablet_service.cc:1467] Processing CreateTablet for tablet 7615406f98fc4059b858e1fa9ab72efe (DEFAULT_TABLE table=test-table [id=e9b6a3205b794dda888e91513a04329b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:12:24.612691 13282 tablet_service.cc:1467] Processing CreateTablet for tablet 7615406f98fc4059b858e1fa9ab72efe (DEFAULT_TABLE table=test-table [id=e9b6a3205b794dda888e91513a04329b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:12:24.613025 13418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7615406f98fc4059b858e1fa9ab72efe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:12:24.613024 13282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7615406f98fc4059b858e1fa9ab72efe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:12:24.616859 13652 tablet_bootstrap.cc:492] T 7615406f98fc4059b858e1fa9ab72efe P 0a4be25da75343968fa10e8f6229d8f9: Bootstrap starting.
I20260926 01:12:24.617439 13652 tablet_bootstrap.cc:654] T 7615406f98fc4059b858e1fa9ab72efe P 0a4be25da75343968fa10e8f6229d8f9: Neither blocks nor log segments found. Creating new log.
I20260926 01:12:24.617697 13652 log.cc:824] T 7615406f98fc4059b858e1fa9ab72efe P 0a4be25da75343968fa10e8f6229d8f9: Log is configured to *not* fsync() on all Append() calls
I20260926 01:12:24.618021 13651 tablet_bootstrap.cc:492] T 7615406f98fc4059b858e1fa9ab72efe P 66b49e280a66492b9f4d6678238efe82: Bootstrap starting.
I20260926 01:12:24.618543 13651 tablet_bootstrap.cc:654] T 7615406f98fc4059b858e1fa9ab72efe P 66b49e280a66492b9f4d6678238efe82: Neither blocks nor log segments found. Creating new log.
I20260926 01:12:24.618813 13651 log.cc:824] T 7615406f98fc4059b858e1fa9ab72efe P 66b49e280a66492b9f4d6678238efe82: Log is configured to *not* fsync() on all Append() calls
I20260926 01:12:24.619493 13651 tablet_bootstrap.cc:492] T 7615406f98fc4059b858e1fa9ab72efe P 66b49e280a66492b9f4d6678238efe82: No bootstrap required, opened a new log
I20260926 01:12:24.619553 13651 ts_tablet_manager.cc:1397] T 7615406f98fc4059b858e1fa9ab72efe P 66b49e280a66492b9f4d6678238efe82: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 01:12:24.621637 13651 raft_consensus.cc:348] T 7615406f98fc4059b858e1fa9ab72efe P 66b49e280a66492b9f4d6678238efe82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a4be25da75343968fa10e8f6229d8f9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44545 } } peers { permanent_uuid: "66b49e280a66492b9f4d6678238efe82" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40123 } } peers { permanent_uuid: "416d2ffd801f4c62a95304d024686028" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33425 } }
I20260926 01:12:24.621783 13651 raft_consensus.cc:374] T 7615406f98fc4059b858e1fa9ab72efe P 66b49e280a66492b9f4d6678238efe82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:12:24.621850 13651 raft_consensus.cc:729] T 7615406f98fc4059b858e1fa9ab72efe P 66b49e280a66492b9f4d6678238efe82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66b49e280a66492b9f4d6678238efe82, State: Initialized, Role: FOLLOWER
I20260926 01:12:24.621973 13651 consensus_queue.cc:260] T 7615406f98fc4059b858e1fa9ab72efe P 66b49e280a66492b9f4d6678238efe82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a4be25da75343968fa10e8f6229d8f9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44545 } } peers { permanent_uuid: "66b49e280a66492b9f4d6678238efe82" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40123 } } peers { permanent_uuid: "416d2ffd801f4c62a95304d024686028" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33425 } }
I20260926 01:12:24.622247 13651 ts_tablet_manager.cc:1428] T 7615406f98fc4059b858e1fa9ab72efe P 66b49e280a66492b9f4d6678238efe82: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 01:12:24.622292 13617 heartbeater.cc:503] Master 127.0.98.62:43267 was elected leader, sending a full tablet report...
I20260926 01:12:24.623500 13653 tablet_bootstrap.cc:492] T 7615406f98fc4059b858e1fa9ab72efe P 416d2ffd801f4c62a95304d024686028: Bootstrap starting.
I20260926 01:12:24.624230 13653 tablet_bootstrap.cc:654] T 7615406f98fc4059b858e1fa9ab72efe P 416d2ffd801f4c62a95304d024686028: Neither blocks nor log segments found. Creating new log.
I20260926 01:12:24.625010 13653 log.cc:824] T 7615406f98fc4059b858e1fa9ab72efe P 416d2ffd801f4c62a95304d024686028: Log is configured to *not* fsync() on all Append() calls
I20260926 01:12:24.625918 13653 tablet_bootstrap.cc:492] T 7615406f98fc4059b858e1fa9ab72efe P 416d2ffd801f4c62a95304d024686028: No bootstrap required, opened a new log
I20260926 01:12:24.625980 13653 ts_tablet_manager.cc:1397] T 7615406f98fc4059b858e1fa9ab72efe P 416d2ffd801f4c62a95304d024686028: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 01:12:24.628508 13653 raft_consensus.cc:348] T 7615406f98fc4059b858e1fa9ab72efe P 416d2ffd801f4c62a95304d024686028 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a4be25da75343968fa10e8f6229d8f9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44545 } } peers { permanent_uuid: "66b49e280a66492b9f4d6678238efe82" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40123 } } peers { permanent_uuid: "416d2ffd801f4c62a95304d024686028" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33425 } }
I20260926 01:12:24.628676 13653 raft_consensus.cc:374] T 7615406f98fc4059b858e1fa9ab72efe P 416d2ffd801f4c62a95304d024686028 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:12:24.628703 13653 raft_consensus.cc:729] T 7615406f98fc4059b858e1fa9ab72efe P 416d2ffd801f4c62a95304d024686028 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 416d2ffd801f4c62a95304d024686028, State: Initialized, Role: FOLLOWER
I20260926 01:12:24.628858 13653 consensus_queue.cc:260] T 7615406f98fc4059b858e1fa9ab72efe P 416d2ffd801f4c62a95304d024686028 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a4be25da75343968fa10e8f6229d8f9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44545 } } peers { permanent_uuid: "66b49e280a66492b9f4d6678238efe82" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40123 } } peers { permanent_uuid: "416d2ffd801f4c62a95304d024686028" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33425 } }
I20260926 01:12:24.629115 13653 ts_tablet_manager.cc:1428] T 7615406f98fc4059b858e1fa9ab72efe P 416d2ffd801f4c62a95304d024686028: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 01:12:24.629184 13484 heartbeater.cc:503] Master 127.0.98.62:43267 was elected leader, sending a full tablet report...
I20260926 01:12:24.629640 13652 tablet_bootstrap.cc:492] T 7615406f98fc4059b858e1fa9ab72efe P 0a4be25da75343968fa10e8f6229d8f9: No bootstrap required, opened a new log
I20260926 01:12:24.629709 13652 ts_tablet_manager.cc:1397] T 7615406f98fc4059b858e1fa9ab72efe P 0a4be25da75343968fa10e8f6229d8f9: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.001s
I20260926 01:12:24.632262 13652 raft_consensus.cc:348] T 7615406f98fc4059b858e1fa9ab72efe P 0a4be25da75343968fa10e8f6229d8f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a4be25da75343968fa10e8f6229d8f9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44545 } } peers { permanent_uuid: "66b49e280a66492b9f4d6678238efe82" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40123 } } peers { permanent_uuid: "416d2ffd801f4c62a95304d024686028" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33425 } }
I20260926 01:12:24.632447 13652 raft_consensus.cc:374] T 7615406f98fc4059b858e1fa9ab72efe P 0a4be25da75343968fa10e8f6229d8f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:12:24.632476 13652 raft_consensus.cc:729] T 7615406f98fc4059b858e1fa9ab72efe P 0a4be25da75343968fa10e8f6229d8f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a4be25da75343968fa10e8f6229d8f9, State: Initialized, Role: FOLLOWER
I20260926 01:12:24.632612 13652 consensus_queue.cc:260] T 7615406f98fc4059b858e1fa9ab72efe P 0a4be25da75343968fa10e8f6229d8f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a4be25da75343968fa10e8f6229d8f9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44545 } } peers { permanent_uuid: "66b49e280a66492b9f4d6678238efe82" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40123 } } peers { permanent_uuid: "416d2ffd801f4c62a95304d024686028" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33425 } }
I20260926 01:12:24.632894 13652 ts_tablet_manager.cc:1428] T 7615406f98fc4059b858e1fa9ab72efe P 0a4be25da75343968fa10e8f6229d8f9: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260926 01:12:24.632951 13350 heartbeater.cc:503] Master 127.0.98.62:43267 was elected leader, sending a full tablet report...
I20260926 01:12:24.636677 13656 raft_consensus.cc:482] T 7615406f98fc4059b858e1fa9ab72efe P 66b49e280a66492b9f4d6678238efe82 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:12:24.636924 13656 raft_consensus.cc:504] T 7615406f98fc4059b858e1fa9ab72efe P 66b49e280a66492b9f4d6678238efe82 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a4be25da75343968fa10e8f6229d8f9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44545 } } peers { permanent_uuid: "66b49e280a66492b9f4d6678238efe82" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40123 } } peers { permanent_uuid: "416d2ffd801f4c62a95304d024686028" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33425 } }
I20260926 01:12:24.637879 13656 leader_election.cc:288] T 7615406f98fc4059b858e1fa9ab72efe P 66b49e280a66492b9f4d6678238efe82 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a4be25da75343968fa10e8f6229d8f9 (127.0.98.1:44545), 416d2ffd801f4c62a95304d024686028 (127.0.98.2:33425)
I20260926 01:12:24.641736 13649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:24.637804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47569 (local address 127.0.98.2:33425)
0926 01:12:24.637870 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 01:12:24.637872 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:12:24.638122 (+ 250us) server_negotiation.cc:407] Connection header received
0926 01:12:24.638195 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:12:24.638196 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:12:24.638208 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:12:24.638276 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:12:24.638517 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:12:24.639340 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:12:24.640762 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:12:24.641367 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:12:24.641412 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:12:24.641423 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 01:12:24.641623 (+ 200us) server_negotiation.cc:299] Negotiation successful
0926 01:12:24.641663 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 01:12:24.642261 13661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:24.637515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33425 (local address 127.0.98.3:47569)
0926 01:12:24.638037 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0926 01:12:24.638051 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 01:12:24.638126 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:12:24.638343 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:12:24.638345 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:12:24.638347 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:12:24.638443 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:12:24.638445 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:12:24.639436 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:12:24.639437 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:12:24.640664 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:12:24.640666 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:12:24.641478 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:12:24.641479 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:12:24.641550 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:12:24.641554 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 01:12:24.642115 (+ 561us) client_negotiation.cc:240] Negotiation successful
0926 01:12:24.642162 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":141,"threads_started":1}
I20260926 01:12:24.642729 13438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7615406f98fc4059b858e1fa9ab72efe" candidate_uuid: "66b49e280a66492b9f4d6678238efe82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "416d2ffd801f4c62a95304d024686028" is_pre_election: true
I20260926 01:12:24.642791 13646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:24.637729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46885 (local address 127.0.98.1:44545)
0926 01:12:24.637781 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 01:12:24.637783 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:12:24.638312 (+ 529us) server_negotiation.cc:407] Connection header received
0926 01:12:24.638413 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:12:24.638414 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:12:24.638427 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:12:24.638459 (+ 32us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:12:24.638711 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:12:24.639674 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:12:24.641530 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:12:24.642197 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:12:24.642250 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:12:24.642261 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 01:12:24.642670 (+ 409us) server_negotiation.cc:299] Negotiation successful
0926 01:12:24.642720 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 01:12:24.642786 13660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:24.637515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44545 (local address 127.0.98.3:46885)
0926 01:12:24.638253 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0926 01:12:24.638262 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 01:12:24.638338 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:12:24.638530 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:12:24.638531 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:12:24.638533 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:12:24.638623 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:12:24.638625 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:12:24.639800 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:12:24.639801 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:12:24.641419 (+ 1618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:12:24.641422 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:12:24.642392 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:12:24.642393 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:12:24.642572 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:12:24.642577 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 01:12:24.642656 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 01:12:24.642696 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":720,"mutex_wait_us":15,"thread_start_us":107,"threads_started":1}
I20260926 01:12:24.642894 13438 raft_consensus.cc:2454] T 7615406f98fc4059b858e1fa9ab72efe P 416d2ffd801f4c62a95304d024686028 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b49e280a66492b9f4d6678238efe82 in term 0.
I20260926 01:12:24.643090 13304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7615406f98fc4059b858e1fa9ab72efe" candidate_uuid: "66b49e280a66492b9f4d6678238efe82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a4be25da75343968fa10e8f6229d8f9" is_pre_election: true
I20260926 01:12:24.643158 13506 leader_election.cc:302] T 7615406f98fc4059b858e1fa9ab72efe P 66b49e280a66492b9f4d6678238efe82 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 416d2ffd801f4c62a95304d024686028, 66b49e280a66492b9f4d6678238efe82; no voters:
I20260926 01:12:24.643215 13304 raft_consensus.cc:2454] T 7615406f98fc4059b858e1fa9ab72efe P 0a4be25da75343968fa10e8f6229d8f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b49e280a66492b9f4d6678238efe82 in term 0.
I20260926 01:12:24.643326 13656 raft_consensus.cc:2781] T 7615406f98fc4059b858e1fa9ab72efe P 66b49e280a66492b9f4d6678238efe82 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:12:24.643378 13656 raft_consensus.cc:482] T 7615406f98fc4059b858e1fa9ab72efe P 66b49e280a66492b9f4d6678238efe82 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:12:24.643400 13656 raft_consensus.cc:3037] T 7615406f98fc4059b858e1fa9ab72efe P 66b49e280a66492b9f4d6678238efe82 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:12:24.644325 13656 raft_consensus.cc:504] T 7615406f98fc4059b858e1fa9ab72efe P 66b49e280a66492b9f4d6678238efe82 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a4be25da75343968fa10e8f6229d8f9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44545 } } peers { permanent_uuid: "66b49e280a66492b9f4d6678238efe82" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40123 } } peers { permanent_uuid: "416d2ffd801f4c62a95304d024686028" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33425 } }
I20260926 01:12:24.644475 13656 leader_election.cc:288] T 7615406f98fc4059b858e1fa9ab72efe P 66b49e280a66492b9f4d6678238efe82 [CANDIDATE]: Term 1 election: Requested vote from peers 0a4be25da75343968fa10e8f6229d8f9 (127.0.98.1:44545), 416d2ffd801f4c62a95304d024686028 (127.0.98.2:33425)
I20260926 01:12:24.644661 13438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7615406f98fc4059b858e1fa9ab72efe" candidate_uuid: "66b49e280a66492b9f4d6678238efe82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "416d2ffd801f4c62a95304d024686028"
I20260926 01:12:24.644742 13438 raft_consensus.cc:3037] T 7615406f98fc4059b858e1fa9ab72efe P 416d2ffd801f4c62a95304d024686028 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:12:24.644732 13304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7615406f98fc4059b858e1fa9ab72efe" candidate_uuid: "66b49e280a66492b9f4d6678238efe82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a4be25da75343968fa10e8f6229d8f9"
I20260926 01:12:24.644804 13304 raft_consensus.cc:3037] T 7615406f98fc4059b858e1fa9ab72efe P 0a4be25da75343968fa10e8f6229d8f9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:12:24.645574 13304 raft_consensus.cc:2454] T 7615406f98fc4059b858e1fa9ab72efe P 0a4be25da75343968fa10e8f6229d8f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66b49e280a66492b9f4d6678238efe82 in term 1.
I20260926 01:12:24.645574 13438 raft_consensus.cc:2454] T 7615406f98fc4059b858e1fa9ab72efe P 416d2ffd801f4c62a95304d024686028 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66b49e280a66492b9f4d6678238efe82 in term 1.
I20260926 01:12:24.645881 13506 leader_election.cc:302] T 7615406f98fc4059b858e1fa9ab72efe P 66b49e280a66492b9f4d6678238efe82 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 416d2ffd801f4c62a95304d024686028, 66b49e280a66492b9f4d6678238efe82; no voters:
I20260926 01:12:24.645995 13656 raft_consensus.cc:2781] T 7615406f98fc4059b858e1fa9ab72efe P 66b49e280a66492b9f4d6678238efe82 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:12:24.646169 13656 raft_consensus.cc:686] T 7615406f98fc4059b858e1fa9ab72efe P 66b49e280a66492b9f4d6678238efe82 [term 1 LEADER]: Becoming Leader. State: Replica: 66b49e280a66492b9f4d6678238efe82, State: Running, Role: LEADER
I20260926 01:12:24.646267 13656 consensus_queue.cc:237] T 7615406f98fc4059b858e1fa9ab72efe P 66b49e280a66492b9f4d6678238efe82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a4be25da75343968fa10e8f6229d8f9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44545 } } peers { permanent_uuid: "66b49e280a66492b9f4d6678238efe82" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40123 } } peers { permanent_uuid: "416d2ffd801f4c62a95304d024686028" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33425 } }
I20260926 01:12:24.647135 13152 catalog_manager.cc:5495] T 7615406f98fc4059b858e1fa9ab72efe P 66b49e280a66492b9f4d6678238efe82 reported cstate change: term changed from 0 to 1, leader changed from <none> to 66b49e280a66492b9f4d6678238efe82 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "66b49e280a66492b9f4d6678238efe82" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a4be25da75343968fa10e8f6229d8f9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66b49e280a66492b9f4d6678238efe82" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "416d2ffd801f4c62a95304d024686028" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33425 } health_report { overall_health: UNKNOWN } } }
W20260926 01:12:24.647390 13485 tablet.cc:2369] T 7615406f98fc4059b858e1fa9ab72efe P 416d2ffd801f4c62a95304d024686028: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 01:12:24.655580 13650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:24.652406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57134 (local address 127.0.98.3:40123)
0926 01:12:24.652465 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 01:12:24.652468 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:12:24.652476 (+ 8us) server_negotiation.cc:407] Connection header received
0926 01:12:24.652509 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:12:24.652510 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:12:24.652530 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:12:24.652600 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:12:24.652783 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:12:24.653514 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:12:24.654658 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:12:24.654967 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:12:24.655032 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:12:24.655215 (+ 183us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:12:24.655347 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:12:24.655423 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 01:12:24.655478 (+ 55us) server_negotiation.cc:299] Negotiation successful
0926 01:12:24.655501 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 01:12:24.657574 13438 raft_consensus.cc:1261] T 7615406f98fc4059b858e1fa9ab72efe P 416d2ffd801f4c62a95304d024686028 [term 1 FOLLOWER]: Refusing update from remote peer 66b49e280a66492b9f4d6678238efe82: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:12:24.657680 13304 raft_consensus.cc:1261] T 7615406f98fc4059b858e1fa9ab72efe P 0a4be25da75343968fa10e8f6229d8f9 [term 1 FOLLOWER]: Refusing update from remote peer 66b49e280a66492b9f4d6678238efe82: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:12:24.658056 13656 consensus_queue.cc:1035] T 7615406f98fc4059b858e1fa9ab72efe P 66b49e280a66492b9f4d6678238efe82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a4be25da75343968fa10e8f6229d8f9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:12:24.658243 13662 consensus_queue.cc:1035] T 7615406f98fc4059b858e1fa9ab72efe P 66b49e280a66492b9f4d6678238efe82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "416d2ffd801f4c62a95304d024686028" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:12:24.664211 13665 mvcc.cc:205] Tried to move back new op lower bound from 7333417552514818048 to 7333417552471740416. Current Snapshot: MvccSnapshot[applied={T|T < 7333417552514818048}]
I20260926 01:12:24.665558 13667 mvcc.cc:205] Tried to move back new op lower bound from 7333417552514818048 to 7333417552471740416. Current Snapshot: MvccSnapshot[applied={T|T < 7333417552514818048}]
I20260926 01:12:24.665874 13666 mvcc.cc:205] Tried to move back new op lower bound from 7333417552514818048 to 7333417552471740416. Current Snapshot: MvccSnapshot[applied={T|T < 7333417552514818048}]
I20260926 01:12:24.667114 13151 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47928:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:12:24.667217 13151 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47928:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:12:24.668021 13151 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f82caa48b6ce457880e82369fd70ed96: Sending DeleteTablet for 3 replicas of tablet 7615406f98fc4059b858e1fa9ab72efe
I20260926 01:12:24.668427 13418 tablet_service.cc:1514] Processing DeleteTablet for tablet 7615406f98fc4059b858e1fa9ab72efe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:12:24 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52250
I20260926 01:12:24.668614 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 13190
I20260926 01:12:24.668730 13551 tablet_service.cc:1514] Processing DeleteTablet for tablet 7615406f98fc4059b858e1fa9ab72efe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:12:24 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57130
I20260926 01:12:24.669193 13675 tablet_replica.cc:330] T 7615406f98fc4059b858e1fa9ab72efe P 66b49e280a66492b9f4d6678238efe82: stopping tablet replica
I20260926 01:12:24.669277 13675 raft_consensus.cc:2229] T 7615406f98fc4059b858e1fa9ab72efe P 66b49e280a66492b9f4d6678238efe82 [term 1 LEADER]: Raft consensus shutting down.
I20260926 01:12:24.669356 13675 raft_consensus.cc:2258] T 7615406f98fc4059b858e1fa9ab72efe P 66b49e280a66492b9f4d6678238efe82 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:12:24.669504 13674 tablet_replica.cc:330] T 7615406f98fc4059b858e1fa9ab72efe P 416d2ffd801f4c62a95304d024686028: stopping tablet replica
I20260926 01:12:24.669597 13674 raft_consensus.cc:2229] T 7615406f98fc4059b858e1fa9ab72efe P 416d2ffd801f4c62a95304d024686028 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:12:24.669687 13674 raft_consensus.cc:2258] T 7615406f98fc4059b858e1fa9ab72efe P 416d2ffd801f4c62a95304d024686028 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:12:24.670219 13674 ts_tablet_manager.cc:1905] T 7615406f98fc4059b858e1fa9ab72efe P 416d2ffd801f4c62a95304d024686028: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:12:24.671940 13674 ts_tablet_manager.cc:1918] T 7615406f98fc4059b858e1fa9ab72efe P 416d2ffd801f4c62a95304d024686028: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:12:24.672039 13674 log.cc:1196] T 7615406f98fc4059b858e1fa9ab72efe P 416d2ffd801f4c62a95304d024686028: Deleting WAL directory at /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/ts-1/wal/wals/7615406f98fc4059b858e1fa9ab72efe
I20260926 01:12:24.672348 13674 ts_tablet_manager.cc:1939] T 7615406f98fc4059b858e1fa9ab72efe P 416d2ffd801f4c62a95304d024686028: Deleting consensus metadata
I20260926 01:12:24.672355 13675 ts_tablet_manager.cc:1905] T 7615406f98fc4059b858e1fa9ab72efe P 66b49e280a66492b9f4d6678238efe82: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:12:24.673177 13138 catalog_manager.cc:4841] TS 416d2ffd801f4c62a95304d024686028 (127.0.98.2:33425): tablet 7615406f98fc4059b858e1fa9ab72efe (table test-table [id=e9b6a3205b794dda888e91513a04329b]) successfully deleted
I20260926 01:12:24.674995 13675 ts_tablet_manager.cc:1918] T 7615406f98fc4059b858e1fa9ab72efe P 66b49e280a66492b9f4d6678238efe82: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:12:24.676306 13675 log.cc:1196] T 7615406f98fc4059b858e1fa9ab72efe P 66b49e280a66492b9f4d6678238efe82: Deleting WAL directory at /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384993574762-392-0/minicluster-data/ts-2/wal/wals/7615406f98fc4059b858e1fa9ab72efe
I20260926 01:12:24.676674 13675 ts_tablet_manager.cc:1939] T 7615406f98fc4059b858e1fa9ab72efe P 66b49e280a66492b9f4d6678238efe82: Deleting consensus metadata
I20260926 01:12:24.677203 13138 catalog_manager.cc:4841] TS 66b49e280a66492b9f4d6678238efe82 (127.0.98.3:40123): tablet 7615406f98fc4059b858e1fa9ab72efe (table test-table [id=e9b6a3205b794dda888e91513a04329b]) successfully deleted
W20260926 01:12:24.679108 13137 connection.cc:535] client connection to 127.0.98.1:44545 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260926 01:12:24.679206 13137 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 01:12:24.679890 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 13354
I20260926 01:12:24.686365 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 13487
I20260926 01:12:24.694357 13645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:24.694197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44545 (local address 127.0.0.1:44230)
0926 01:12:24.694259 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0926 01:12:24.694307 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260926 01:12:24.694650 13137 catalog_manager.cc:4584] TS 0a4be25da75343968fa10e8f6229d8f9 (127.0.98.1:44545): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7615406f98fc4059b858e1fa9ab72efe: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44545: connect: Connection refused (error 111)
I20260926 01:12:24.695086 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 13119
I20260926 01:12:24.702051 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260926 01:12:25.036710 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260926 01:12:25.036808 392 mini_ranger.cc:66] Stopping Ranger...
I20260926 01:12:25.371750 392 mini_ranger.cc:68] Stopped Ranger
2026-09-26 01:12:25.371 UTC [12185] LOG: received smart shutdown request
2026-09-26 01:12:25.375 UTC [12185] LOG: background worker "logical replication launcher" (PID 12194) exited with exit code 1
2026-09-26 01:12:25.375 UTC [12189] LOG: shutting down
2026-09-26 01:12:25.554 UTC [12185] LOG: database system is shut down
2026-09-26T01:12:25Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (49342 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384993574762-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 01:12:25 dist-test-slave-7b22 krb5kdc[13685](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 01:12:25 dist-test-slave-7b22 krb5kdc[13685](info): set up 2 sockets
Sep 26 01:12:25 dist-test-slave-7b22 krb5kdc[13685](info): commencing operation
krb5kdc: starting...
W20260926 01:12:26.609498 392 mini_kdc.cc:121] Time spent starting KDC: real 0.944s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384993574762-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384993574762-392-0/krb5kdc/test-user.keytab.
Sep 26 01:12:26 dist-test-slave-7b22 krb5kdc[13685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385146, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T01:12:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:12:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384993574762-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384993574762-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 01:12:26.733873 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384993574762-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384993574762-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384993574762-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33013
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384993574762-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33013,127.0.98.61:43779,127.0.98.60:34667
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:12:26.815084 13698 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:12:26.815331 13698 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:12:26.815364 13698 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:12:26.816636 13698 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:12:26.816691 13698 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:12:26.816709 13698 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:12:26.816725 13698 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:12:26.816741 13698 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:12:26.818434 13698 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384993574762-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384993574762-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33013,127.0.98.61:43779,127.0.98.60:34667
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384993574762-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33013
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13698
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384993574762-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:12:26.818707 13698 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:12:26.818928 13698 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:12:26.823722 13706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:12:26.823719 13704 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:12:26.823719 13703 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:12:26.824038 13698 server_base.cc:1034] running on GCE node
I20260926 01:12:26.824282 13698 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:12:26.824589 13698 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:12:26.825750 13698 hybrid_clock.cc:648] HybridClock initialized: now 1790385146825731 us; error 30 us; skew 500 ppm
Sep 26 01:12:26 dist-test-slave-7b22 krb5kdc[13685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385146, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:12:26.828274 13698 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 01:12:26.828755 13698 webserver.cc:460] Webserver started at http://127.0.98.62:41549/ using document root <none> and password file <none>
I20260926 01:12:26.828975 13698 fs_manager.cc:359] Metadata directory not provided
I20260926 01:12:26.829025 13698 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:12:26.829150 13698 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:12:26.830276 13698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384993574762-392-0/minicluster-data/master-0/data/instance:
uuid: "94da24713c634743a04ecc86dfba3145"
format_stamp: "Formatted at 2026-09-26 01:12:26 on dist-test-slave-7b22"
server_key: "9a86d9e8d00accec8088f6066d2a64c3"
server_key_iv: "bf2ba3d26ec9528011b1e56a6165bbcf"
server_key_version: "encryptionkey@0"
I20260926 01:12:26.830988 13698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384993574762-392-0/minicluster-data/master-0/wal/instance:
uuid: "94da24713c634743a04ecc86dfba3145"
format_stamp: "Formatted at 2026-09-26 01:12:26 on dist-test-slave-7b22"
server_key: "9a86d9e8d00accec8088f6066d2a64c3"
server_key_iv: "bf2ba3d26ec9528011b1e56a6165bbcf"
server_key_version: "encryptionkey@0"
I20260926 01:12:26.833009 13698 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260926 01:12:26.834270 13713 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:12:26.834560 13698 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:12:26.834690 13698 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384993574762-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384993574762-392-0/minicluster-data/master-0/wal
uuid: "94da24713c634743a04ecc86dfba3145"
format_stamp: "Formatted at 2026-09-26 01:12:26 on dist-test-slave-7b22"
server_key: "9a86d9e8d00accec8088f6066d2a64c3"
server_key_iv: "bf2ba3d26ec9528011b1e56a6165bbcf"
server_key_version: "encryptionkey@0"
I20260926 01:12:26.834787 13698 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384993574762-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384993574762-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384993574762-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:12:26.845229 13698 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:12:26.847967 13698 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:12:26.848129 13698 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:12:26.852900 13698 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33013
I20260926 01:12:26.852967 13765 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33013 every 8 connection(s)
I20260926 01:12:26.853332 13698 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
I20260926 01:12:26.854315 13766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:12:26.855957 13766 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33013 } has no permanent_uuid. Determining permanent_uuid...
I20260926 01:12:26.858978 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 13698
I20260926 01:12:26.859079 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384993574762-392-0/minicluster-data/master-0/wal/instance
I20260926 01:12:26.859340 392 external_mini_cluster.cc:1428] Setting key b0acf3c2fa20e6c6aaa2dc2c47004ee9
Sep 26 01:12:26 dist-test-slave-7b22 krb5kdc[13685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385146, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:12:26.867689 13769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:26.856746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33030 (local address 127.0.98.62:33013)
0926 01:12:26.856887 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0926 01:12:26.856897 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 01:12:26.857010 (+ 113us) server_negotiation.cc:407] Connection header received
0926 01:12:26.857544 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:12:26.857546 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:12:26.857564 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:12:26.857656 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:12:26.858329 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:12:26.859263 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:12:26.860635 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:12:26.861357 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:12:26.861488 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:12:26.864090 (+ 2602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:12:26.864098 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:12:26.864099 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:12:26.864106 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:12:26.865873 (+ 1767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:12:26.866250 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:12:26.866254 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:12:26.866255 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:12:26.866417 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:12:26.866645 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:12:26.866646 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:12:26.866646 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:12:26.866971 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:12:26.867059 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 01:12:26.867276 (+ 217us) server_negotiation.cc:299] Negotiation successful
0926 01:12:26.867420 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":69,"threads_started":1}
I20260926 01:12:26.867689 13768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:26.856678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33013 (local address 127.0.0.1:33030)
0926 01:12:26.856870 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0926 01:12:26.856911 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 01:12:26.857388 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:12:26.857702 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:12:26.857705 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:12:26.858065 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:12:26.858229 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:12:26.858234 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:12:26.859373 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:12:26.859374 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:12:26.860408 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:12:26.860411 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:12:26.861465 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:12:26.861466 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:12:26.861562 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:12:26.862218 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:12:26.862228 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:12:26.863978 (+ 1750us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:12:26.865955 (+ 1977us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:12:26.865961 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:12:26.865963 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:12:26.866168 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:12:26.866492 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:12:26.866493 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:12:26.866493 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:12:26.866583 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:12:26.867069 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:12:26.867075 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:12:26.867140 (+ 65us) client_negotiation.cc:769] Sending connection context
0926 01:12:26.867228 (+ 88us) client_negotiation.cc:240] Negotiation successful
0926 01:12:26.867493 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"mutex_wait_us":108,"thread_start_us":83,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
I20260926 01:12:26.868590 13766 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43779 } has no permanent_uuid. Determining permanent_uuid...
I20260926 01:12:26.868904 13768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:26.868799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43779 (local address 127.0.0.1:59514)
0926 01:12:26.868830 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0926 01:12:26.868880 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260926 01:12:26.869087 13715 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43779: connect: Connection refused (error 111)
I20260926 01:12:26.869474 13768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:26.869368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43779 (local address 127.0.0.1:59522)
0926 01:12:26.869426 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0926 01:12:26.869454 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260926 01:12:26.869583 13766 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:43779: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43779: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384993574762-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384993574762-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260926 01:12:26.885347 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384993574762-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384993574762-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384993574762-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384993574762-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:43779
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384993574762-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33013,127.0.98.61:43779,127.0.98.60:34667
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 01:12:26.918820 13766 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43779 } attempt: 1
I20260926 01:12:26.919337 13768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:26.919189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43779 (local address 127.0.0.1:59538)
0926 01:12:26.919246 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0926 01:12:26.919297 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260926 01:12:26.919659 13768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:26.919575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43779 (local address 127.0.0.1:59552)
0926 01:12:26.919614 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0926 01:12:26.919639 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260926 01:12:26.919766 13766 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:43779: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43779: connect: Connection refused (error 111)
W20260926 01:12:26.964383 13773 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:12:26.964596 13773 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:12:26.964617 13773 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:12:26.965793 13773 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:12:26.965951 13773 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:12:26.965987 13773 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:12:26.966001 13773 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:12:26.966015 13773 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:12:26.967787 13773 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384993574762-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384993574762-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33013,127.0.98.61:43779,127.0.98.60:34667
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384993574762-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:43779
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384993574762-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13773
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384993574762-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:12:26.968027 13773 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:12:26.968240 13773 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:12:26.972930 13779 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:12:26.972920 13781 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:12:26.972891 13778 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:12:26.973062 13773 server_base.cc:1034] running on GCE node
I20260926 01:12:26.973367 13773 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:12:26.973654 13773 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:12:26.974835 13773 hybrid_clock.cc:648] HybridClock initialized: now 1790385146974800 us; error 51 us; skew 500 ppm
Sep 26 01:12:26 dist-test-slave-7b22 krb5kdc[13685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385146, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:12:26.977304 13773 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260926 01:12:26.977707 13773 webserver.cc:460] Webserver started at http://127.0.98.61:43645/ using document root <none> and password file <none>
I20260926 01:12:26.977952 13773 fs_manager.cc:359] Metadata directory not provided
I20260926 01:12:26.978001 13773 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:12:26.978122 13773 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:12:26.979189 13773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384993574762-392-0/minicluster-data/master-1/data/instance:
uuid: "272ffbb71f304dedb633e7eb29e42513"
format_stamp: "Formatted at 2026-09-26 01:12:26 on dist-test-slave-7b22"
server_key: "3f367756a1e3d14e5d460472f9c886ed"
server_key_iv: "f944a944e7f62c5bacae4590f505bc76"
server_key_version: "encryptionkey@0"
I20260926 01:12:26.979513 13773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384993574762-392-0/minicluster-data/master-1/wal/instance:
uuid: "272ffbb71f304dedb633e7eb29e42513"
format_stamp: "Formatted at 2026-09-26 01:12:26 on dist-test-slave-7b22"
server_key: "3f367756a1e3d14e5d460472f9c886ed"
server_key_iv: "f944a944e7f62c5bacae4590f505bc76"
server_key_version: "encryptionkey@0"
I20260926 01:12:26.980762 13773 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 01:12:26.981577 13788 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:12:26.981751 13773 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:12:26.981850 13773 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384993574762-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384993574762-392-0/minicluster-data/master-1/wal
uuid: "272ffbb71f304dedb633e7eb29e42513"
format_stamp: "Formatted at 2026-09-26 01:12:26 on dist-test-slave-7b22"
server_key: "3f367756a1e3d14e5d460472f9c886ed"
server_key_iv: "f944a944e7f62c5bacae4590f505bc76"
server_key_version: "encryptionkey@0"
I20260926 01:12:26.981957 13773 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384993574762-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384993574762-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384993574762-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:12:26.987922 13766 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43779 } attempt: 2
I20260926 01:12:26.988355 13768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:26.988239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43779 (local address 127.0.0.1:59562)
0926 01:12:26.988294 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0926 01:12:26.988324 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260926 01:12:26.988674 13768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:26.988609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43779 (local address 127.0.0.1:59564)
0926 01:12:26.988639 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0926 01:12:26.988660 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260926 01:12:26.988744 13766 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:43779: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43779: connect: Connection refused (error 111)
I20260926 01:12:27.003759 13773 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:12:27.005954 13773 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:12:27.006067 13773 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:12:27.009740 13773 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:43779
I20260926 01:12:27.009778 13840 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:43779 every 8 connection(s)
I20260926 01:12:27.010227 13773 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384993574762-392-0/minicluster-data/master-1/data/info.pb
I20260926 01:12:27.011077 13841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:12:27.012600 13841 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33013 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 01:12:27 dist-test-slave-7b22 krb5kdc[13685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385146, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:12:27.020397 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 13773
I20260926 01:12:27.020516 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384993574762-392-0/minicluster-data/master-1/wal/instance
I20260926 01:12:27.020694 392 external_mini_cluster.cc:1428] Setting key 151c5d7c8bc9fb64776c2e58d3e2acc7
I20260926 01:12:27.021384 13769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:27.013325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33038 (local address 127.0.98.62:33013)
0926 01:12:27.013384 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 01:12:27.013387 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:12:27.013547 (+ 160us) server_negotiation.cc:407] Connection header received
0926 01:12:27.014089 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:12:27.014091 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:12:27.014106 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:12:27.014159 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:12:27.014769 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:12:27.015247 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:12:27.016189 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:12:27.016397 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:12:27.016452 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:12:27.018598 (+ 2146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:12:27.018600 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:12:27.018601 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:12:27.018608 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:12:27.019949 (+ 1341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:12:27.020438 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:12:27.020439 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:12:27.020440 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:12:27.020493 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:12:27.020731 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:12:27.020732 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:12:27.020732 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:12:27.020868 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:12:27.020925 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 01:12:27.021194 (+ 269us) server_negotiation.cc:299] Negotiation successful
0926 01:12:27.021295 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 01:12:27.021576 13843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:27.013255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33013 (local address 127.0.0.1:33038)
0926 01:12:27.013427 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0926 01:12:27.013469 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 01:12:27.013997 (+ 528us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:12:27.014237 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:12:27.014241 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:12:27.014487 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:12:27.014683 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:12:27.014687 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:12:27.015323 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:12:27.015324 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:12:27.016112 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:12:27.016114 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:12:27.016467 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:12:27.016468 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:12:27.016587 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:12:27.017063 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:12:27.017071 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:12:27.018503 (+ 1432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:12:27.020043 (+ 1540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:12:27.020052 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:12:27.020054 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:12:27.020369 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:12:27.020549 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:12:27.020549 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:12:27.020550 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:12:27.020674 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:12:27.020923 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:12:27.020938 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:12:27.021078 (+ 140us) client_negotiation.cc:769] Sending connection context
0926 01:12:27.021168 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 01:12:27.021337 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":76,"threads_started":1}
I20260926 01:12:27.022368 13841 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43779 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 01:12:27 dist-test-slave-7b22 krb5kdc[13685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385146, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260926 01:12:27.030903 13843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:27.022625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43779 (local address 127.0.0.1:59568)
0926 01:12:27.022676 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0926 01:12:27.022688 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 01:12:27.022731 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:12:27.023022 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:12:27.023023 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:12:27.023131 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:12:27.023185 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:12:27.023186 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:12:27.024391 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:12:27.024392 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:12:27.024916 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:12:27.024918 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:12:27.025563 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:12:27.025568 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:12:27.025633 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:12:27.026142 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:12:27.026147 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:12:27.027547 (+ 1400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:12:27.029492 (+ 1945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:12:27.029493 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:12:27.029494 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:12:27.029689 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:12:27.029984 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:12:27.029985 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:12:27.029985 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:12:27.030201 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:12:27.030572 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:12:27.030574 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:12:27.030633 (+ 59us) client_negotiation.cc:769] Sending connection context
0926 01:12:27.030709 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 01:12:27.030823 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260926 01:12:27.031011 13845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:27.022679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59568 (local address 127.0.98.61:43779)
0926 01:12:27.022838 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0926 01:12:27.022843 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 01:12:27.022863 (+ 20us) server_negotiation.cc:407] Connection header received
0926 01:12:27.022885 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:12:27.022887 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:12:27.022900 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:12:27.023008 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:12:27.023242 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:12:27.024291 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:12:27.025000 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:12:27.025479 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:12:27.025556 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:12:27.027641 (+ 2085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:12:27.027653 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:12:27.027654 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:12:27.027662 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:12:27.029394 (+ 1732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:12:27.029799 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:12:27.029802 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:12:27.029803 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:12:27.029903 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:12:27.030293 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:12:27.030294 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:12:27.030294 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:12:27.030477 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:12:27.030545 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 01:12:27.030778 (+ 233us) server_negotiation.cc:299] Negotiation successful
0926 01:12:27.030900 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":128,"thread_start_us":87,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
I20260926 01:12:27.031751 13841 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34667 } has no permanent_uuid. Determining permanent_uuid...
I20260926 01:12:27.032172 13843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:27.032054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34667 (local address 127.0.0.1:51208)
0926 01:12:27.032099 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0926 01:12:27.032147 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260926 01:12:27.032300 13790 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34667: connect: Connection refused (error 111)
I20260926 01:12:27.032733 13843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:27.032638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34667 (local address 127.0.0.1:51222)
0926 01:12:27.032676 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0926 01:12:27.032712 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260926 01:12:27.032920 13841 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34667: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34667: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384993574762-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384993574762-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260926 01:12:27.050009 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384993574762-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384993574762-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384993574762-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384993574762-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:34667
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384993574762-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33013,127.0.98.61:43779,127.0.98.60:34667
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 01:12:27.079916 13766 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43779 } attempt: 3
I20260926 01:12:27.082186 13841 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34667 } attempt: 1
I20260926 01:12:27.082639 13843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:27.082477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34667 (local address 127.0.0.1:51230)
0926 01:12:27.082553 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0926 01:12:27.082600 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260926 01:12:27.083323 13843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:27.083234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34667 (local address 127.0.0.1:51242)
0926 01:12:27.083267 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0926 01:12:27.083293 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
W20260926 01:12:27.083423 13841 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34667: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34667: connect: Connection refused (error 111)
Sep 26 01:12:27 dist-test-slave-7b22 krb5kdc[13685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385146, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260926 01:12:27.087963 13845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:27.080420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59584 (local address 127.0.98.61:43779)
0926 01:12:27.080492 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0926 01:12:27.080494 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:12:27.080506 (+ 12us) server_negotiation.cc:407] Connection header received
0926 01:12:27.080538 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:12:27.080539 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:12:27.080553 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:12:27.080614 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:12:27.080942 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:12:27.081735 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:12:27.082675 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:12:27.082935 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:12:27.082998 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:12:27.085316 (+ 2318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:12:27.085318 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:12:27.085318 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:12:27.085326 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:12:27.086918 (+ 1592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:12:27.087250 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:12:27.087252 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:12:27.087252 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:12:27.087309 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:12:27.087473 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:12:27.087474 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:12:27.087475 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:12:27.087591 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:12:27.087646 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 01:12:27.087782 (+ 136us) server_negotiation.cc:299] Negotiation successful
0926 01:12:27.087865 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 01:12:27.088014 13768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:27.080342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43779 (local address 127.0.0.1:59584)
0926 01:12:27.080414 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0926 01:12:27.080428 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 01:12:27.080494 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:12:27.080653 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:12:27.080654 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:12:27.080797 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:12:27.080867 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:12:27.080868 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:12:27.081879 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:12:27.081880 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:12:27.082606 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:12:27.082608 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:12:27.083003 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:12:27.083004 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:12:27.083077 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:12:27.083552 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:12:27.083559 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:12:27.085212 (+ 1653us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:12:27.087015 (+ 1803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:12:27.087016 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:12:27.087017 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:12:27.087185 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:12:27.087363 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:12:27.087364 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:12:27.087364 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:12:27.087416 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:12:27.087648 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:12:27.087649 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:12:27.087697 (+ 48us) client_negotiation.cc:769] Sending connection context
0926 01:12:27.087751 (+ 54us) client_negotiation.cc:240] Negotiation successful
0926 01:12:27.087863 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260926 01:12:27.088434 13766 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34667 } has no permanent_uuid. Determining permanent_uuid...
I20260926 01:12:27.088759 13768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:27.088669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34667 (local address 127.0.0.1:51252)
0926 01:12:27.088709 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0926 01:12:27.088735 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260926 01:12:27.089048 13768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:27.088981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34667 (local address 127.0.0.1:51256)
0926 01:12:27.089014 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0926 01:12:27.089035 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260926 01:12:27.089125 13766 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34667: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34667: connect: Connection refused (error 111)
I20260926 01:12:27.120317 13766 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34667 } attempt: 1
I20260926 01:12:27.120824 13768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:27.120679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34667 (local address 127.0.0.1:51258)
0926 01:12:27.120745 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0926 01:12:27.120789 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260926 01:12:27.121152 13768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:27.121052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34667 (local address 127.0.0.1:51268)
0926 01:12:27.121097 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0926 01:12:27.121129 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260926 01:12:27.121237 13766 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34667: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34667: connect: Connection refused (error 111)
W20260926 01:12:27.133800 13848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:12:27.134073 13848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:12:27.134096 13848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:12:27.135291 13848 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:12:27.135334 13848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:12:27.135349 13848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:12:27.135363 13848 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:12:27.135375 13848 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:12:27.136922 13848 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384993574762-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384993574762-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33013,127.0.98.61:43779,127.0.98.60:34667
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384993574762-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:34667
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384993574762-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13848
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384993574762-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:12:27.137156 13848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:12:27.137382 13848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:12:27.142105 13856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:12:27.142140 13854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:12:27.142225 13853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:12:27.142117 13848 server_base.cc:1034] running on GCE node
I20260926 01:12:27.142515 13848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:12:27.142787 13848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:12:27.143940 13848 hybrid_clock.cc:648] HybridClock initialized: now 1790385147143921 us; error 34 us; skew 500 ppm
Sep 26 01:12:27 dist-test-slave-7b22 krb5kdc[13685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385147, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:12:27.146288 13848 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260926 01:12:27.146763 13848 webserver.cc:460] Webserver started at http://127.0.98.60:36971/ using document root <none> and password file <none>
I20260926 01:12:27.146978 13848 fs_manager.cc:359] Metadata directory not provided
I20260926 01:12:27.147025 13848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:12:27.147148 13848 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:12:27.148196 13848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384993574762-392-0/minicluster-data/master-2/data/instance:
uuid: "804353922ef94f73b5a375e74d61ba1b"
format_stamp: "Formatted at 2026-09-26 01:12:27 on dist-test-slave-7b22"
server_key: "13e7525e51d1fe9e59948928e53df20c"
server_key_iv: "1fd1deb701b72abdd3052b2bf3135f2d"
server_key_version: "encryptionkey@0"
I20260926 01:12:27.148538 13848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384993574762-392-0/minicluster-data/master-2/wal/instance:
uuid: "804353922ef94f73b5a375e74d61ba1b"
format_stamp: "Formatted at 2026-09-26 01:12:27 on dist-test-slave-7b22"
server_key: "13e7525e51d1fe9e59948928e53df20c"
server_key_iv: "1fd1deb701b72abdd3052b2bf3135f2d"
server_key_version: "encryptionkey@0"
I20260926 01:12:27.149725 13848 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 01:12:27.150415 13863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:12:27.150568 13848 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 01:12:27.150652 13848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384993574762-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384993574762-392-0/minicluster-data/master-2/wal
uuid: "804353922ef94f73b5a375e74d61ba1b"
format_stamp: "Formatted at 2026-09-26 01:12:27 on dist-test-slave-7b22"
server_key: "13e7525e51d1fe9e59948928e53df20c"
server_key_iv: "1fd1deb701b72abdd3052b2bf3135f2d"
server_key_version: "encryptionkey@0"
I20260926 01:12:27.150717 13848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384993574762-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384993574762-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384993574762-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:12:27.151584 13841 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34667 } attempt: 2
I20260926 01:12:27.151950 13843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:27.151839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34667 (local address 127.0.0.1:51274)
0926 01:12:27.151887 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0926 01:12:27.151917 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260926 01:12:27.152248 13843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:27.152163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34667 (local address 127.0.0.1:51276)
0926 01:12:27.152201 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0926 01:12:27.152224 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260926 01:12:27.152319 13841 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34667: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34667: connect: Connection refused (error 111)
I20260926 01:12:27.161383 13848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:12:27.163519 13848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:12:27.163659 13848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:12:27.167543 13848 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:34667
I20260926 01:12:27.167577 13915 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:34667 every 8 connection(s)
I20260926 01:12:27.167899 13848 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384993574762-392-0/minicluster-data/master-2/data/info.pb
I20260926 01:12:27.168699 13916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:12:27.170414 13916 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33013 } has no permanent_uuid. Determining permanent_uuid...
I20260926 01:12:27.174923 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 13848
I20260926 01:12:27.175067 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384993574762-392-0/minicluster-data/master-2/wal/instance
I20260926 01:12:27.175254 392 external_mini_cluster.cc:1428] Setting key 39cd78747bfbd4b473bea302cf17d826
Sep 26 01:12:27 dist-test-slave-7b22 krb5kdc[13685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385147, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:12:27.179271 13769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:27.171163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33042 (local address 127.0.98.62:33013)
0926 01:12:27.171228 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 01:12:27.171230 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:12:27.171397 (+ 167us) server_negotiation.cc:407] Connection header received
0926 01:12:27.171833 (+ 436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:12:27.171834 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:12:27.171843 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:12:27.171917 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:12:27.172459 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:12:27.172952 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:12:27.173935 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:12:27.174121 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:12:27.174180 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:12:27.176424 (+ 2244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:12:27.176426 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:12:27.176426 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:12:27.176434 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:12:27.177920 (+ 1486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:12:27.178275 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:12:27.178276 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:12:27.178277 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:12:27.178332 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:12:27.178555 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:12:27.178556 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:12:27.178556 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:12:27.178667 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:12:27.178722 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 01:12:27.178966 (+ 244us) server_negotiation.cc:299] Negotiation successful
0926 01:12:27.179046 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 01:12:27.179312 13918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:27.171122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33013 (local address 127.0.0.1:33042)
0926 01:12:27.171282 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 01:12:27.171317 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 01:12:27.171756 (+ 439us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:12:27.171960 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:12:27.171963 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:12:27.172202 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:12:27.172397 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:12:27.172400 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:12:27.173016 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:12:27.173017 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:12:27.173802 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:12:27.173804 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:12:27.174192 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:12:27.174193 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:12:27.174291 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:12:27.174798 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:12:27.174808 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:12:27.176340 (+ 1532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:12:27.177986 (+ 1646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:12:27.177992 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:12:27.177994 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:12:27.178203 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:12:27.178386 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:12:27.178387 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:12:27.178387 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:12:27.178498 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:12:27.178719 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:12:27.178725 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:12:27.178850 (+ 125us) client_negotiation.cc:769] Sending connection context
0926 01:12:27.178937 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 01:12:27.179096 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":83,"threads_started":1}
I20260926 01:12:27.179997 13916 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43779 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 01:12:27 dist-test-slave-7b22 krb5kdc[13685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385146, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260926 01:12:27.183465 13920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:27.175621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51290 (local address 127.0.98.60:34667)
0926 01:12:27.175767 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0926 01:12:27.175772 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 01:12:27.175799 (+ 27us) server_negotiation.cc:407] Connection header received
0926 01:12:27.175892 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:12:27.175896 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:12:27.175914 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:12:27.175973 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:12:27.176577 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:12:27.177271 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:12:27.178237 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:12:27.178548 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:12:27.178605 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:12:27.180841 (+ 2236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:12:27.180847 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:12:27.180848 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:12:27.180853 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:12:27.182272 (+ 1419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:12:27.182655 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:12:27.182658 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:12:27.182658 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:12:27.182720 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:12:27.182895 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:12:27.182895 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:12:27.182896 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:12:27.183030 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:12:27.183145 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 01:12:27.183251 (+ 106us) server_negotiation.cc:299] Negotiation successful
0926 01:12:27.183352 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":76,"threads_started":1}
Sep 26 01:12:27 dist-test-slave-7b22 krb5kdc[13685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385147, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260926 01:12:27.187103 13918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:27.180566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43779 (local address 127.0.0.1:59594)
0926 01:12:27.180613 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0926 01:12:27.180623 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 01:12:27.180656 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:12:27.180756 (+ 100us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:12:27.180757 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:12:27.180900 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:12:27.180947 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:12:27.180949 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:12:27.181797 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:12:27.181798 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:12:27.182411 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:12:27.182413 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:12:27.182784 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:12:27.182785 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:12:27.182861 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:12:27.183224 (+ 363us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:12:27.183228 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:12:27.184566 (+ 1338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:12:27.186162 (+ 1596us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:12:27.186163 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:12:27.186164 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:12:27.186372 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:12:27.186552 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:12:27.186552 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:12:27.186553 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:12:27.186610 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:12:27.186844 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:12:27.186845 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:12:27.186883 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 01:12:27.186928 (+ 45us) client_negotiation.cc:240] Negotiation successful
0926 01:12:27.187004 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260926 01:12:27.187152 13845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:27.180622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59594 (local address 127.0.98.61:43779)
0926 01:12:27.180664 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 01:12:27.180665 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 01:12:27.180675 (+ 10us) server_negotiation.cc:407] Connection header received
0926 01:12:27.180699 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:12:27.180700 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:12:27.180707 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:12:27.180749 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:12:27.181026 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:12:27.181703 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:12:27.182504 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:12:27.182722 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:12:27.182784 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:12:27.184646 (+ 1862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:12:27.184647 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:12:27.184648 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:12:27.184654 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:12:27.186094 (+ 1440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:12:27.186447 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:12:27.186447 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:12:27.186448 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:12:27.186501 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:12:27.186667 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:12:27.186668 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:12:27.186668 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:12:27.186791 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:12:27.186850 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 01:12:27.186959 (+ 109us) server_negotiation.cc:299] Negotiation successful
0926 01:12:27.187068 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 01:12:27.187592 13916 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34667 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 01:12:27 dist-test-slave-7b22 krb5kdc[13685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385146, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 26 01:12:27 dist-test-slave-7b22 krb5kdc[13685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385147, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260926 01:12:27.191752 13769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:27.184245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33054 (local address 127.0.98.62:33013)
0926 01:12:27.184317 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0926 01:12:27.184319 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:12:27.184343 (+ 24us) server_negotiation.cc:407] Connection header received
0926 01:12:27.184467 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:12:27.184467 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:12:27.184477 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:12:27.184527 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:12:27.185066 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:12:27.185803 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:12:27.186695 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:12:27.186969 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:12:27.187030 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:12:27.189205 (+ 2175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:12:27.189207 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:12:27.189207 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:12:27.189214 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:12:27.190636 (+ 1422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:12:27.190956 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:12:27.190958 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:12:27.190958 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:12:27.191028 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:12:27.191184 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:12:27.191184 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:12:27.191185 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:12:27.191313 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:12:27.191445 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0926 01:12:27.191576 (+ 131us) server_negotiation.cc:299] Negotiation successful
0926 01:12:27.191659 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 01:12:27.194041 13918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:27.187821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34667 (local address 127.0.0.1:51304)
0926 01:12:27.187936 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0926 01:12:27.187946 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 01:12:27.188000 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:12:27.188105 (+ 105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:12:27.188106 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:12:27.188230 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:12:27.188269 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:12:27.188271 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:12:27.188867 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:12:27.188868 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:12:27.189432 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:12:27.189434 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:12:27.189914 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:12:27.189915 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:12:27.189967 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:12:27.190335 (+ 368us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:12:27.190339 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:12:27.191465 (+ 1126us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:12:27.193109 (+ 1644us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:12:27.193110 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:12:27.193111 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:12:27.193308 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:12:27.193474 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:12:27.193475 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:12:27.193475 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:12:27.193535 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:12:27.193722 (+ 187us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:12:27.193723 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:12:27.193764 (+ 41us) client_negotiation.cc:769] Sending connection context
0926 01:12:27.193853 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 01:12:27.193971 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99}
I20260926 01:12:27.194130 13920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:27.187890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51304 (local address 127.0.98.60:34667)
0926 01:12:27.187929 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 01:12:27.187931 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:12:27.187987 (+ 56us) server_negotiation.cc:407] Connection header received
0926 01:12:27.188053 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:12:27.188053 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:12:27.188062 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:12:27.188099 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:12:27.188328 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:12:27.188796 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:12:27.189508 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:12:27.189793 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:12:27.189888 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:12:27.191536 (+ 1648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:12:27.191538 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:12:27.191538 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:12:27.191545 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:12:27.193027 (+ 1482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:12:27.193373 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:12:27.193374 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:12:27.193374 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:12:27.193421 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:12:27.193588 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:12:27.193588 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:12:27.193589 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:12:27.193679 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:12:27.193720 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0926 01:12:27.193870 (+ 150us) server_negotiation.cc:299] Negotiation successful
0926 01:12:27.194027 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":78,"server-negotiator.queue_time_us":22}
I20260926 01:12:27.195744 13916 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 804353922ef94f73b5a375e74d61ba1b: Bootstrap starting.
I20260926 01:12:27.196368 13766 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34667 } attempt: 2
I20260926 01:12:27.196777 13916 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 804353922ef94f73b5a375e74d61ba1b: Neither blocks nor log segments found. Creating new log.
Sep 26 01:12:27 dist-test-slave-7b22 krb5kdc[13685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385146, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260926 01:12:27.197070 13916 log.cc:824] T 00000000000000000000000000000000 P 804353922ef94f73b5a375e74d61ba1b: Log is configured to *not* fsync() on all Append() calls
I20260926 01:12:27.197964 13916 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 804353922ef94f73b5a375e74d61ba1b: No bootstrap required, opened a new log
I20260926 01:12:27.199564 13916 raft_consensus.cc:348] T 00000000000000000000000000000000 P 804353922ef94f73b5a375e74d61ba1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da24713c634743a04ecc86dfba3145" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33013 } } peers { permanent_uuid: "272ffbb71f304dedb633e7eb29e42513" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43779 } } peers { permanent_uuid: "804353922ef94f73b5a375e74d61ba1b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34667 } }
I20260926 01:12:27.199725 13916 raft_consensus.cc:374] T 00000000000000000000000000000000 P 804353922ef94f73b5a375e74d61ba1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:12:27.199765 13916 raft_consensus.cc:729] T 00000000000000000000000000000000 P 804353922ef94f73b5a375e74d61ba1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 804353922ef94f73b5a375e74d61ba1b, State: Initialized, Role: FOLLOWER
I20260926 01:12:27.199889 13916 consensus_queue.cc:260] T 00000000000000000000000000000000 P 804353922ef94f73b5a375e74d61ba1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da24713c634743a04ecc86dfba3145" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33013 } } peers { permanent_uuid: "272ffbb71f304dedb633e7eb29e42513" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43779 } } peers { permanent_uuid: "804353922ef94f73b5a375e74d61ba1b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34667 } }
I20260926 01:12:27.200219 13916 sys_catalog.cc:564] T 00000000000000000000000000000000 P 804353922ef94f73b5a375e74d61ba1b [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:12:27.200558 13845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:27.192471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59608 (local address 127.0.98.61:43779)
0926 01:12:27.192517 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 01:12:27.192519 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:12:27.192527 (+ 8us) server_negotiation.cc:407] Connection header received
0926 01:12:27.192551 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:12:27.192552 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:12:27.192561 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:12:27.192599 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:12:27.193154 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:12:27.193871 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:12:27.194709 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:12:27.194885 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:12:27.194945 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:12:27.197479 (+ 2534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:12:27.197480 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:12:27.197481 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:12:27.197490 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:12:27.199065 (+ 1575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:12:27.199444 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:12:27.199446 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:12:27.199446 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:12:27.199513 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:12:27.199682 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:12:27.199683 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:12:27.199683 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:12:27.200037 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:12:27.200195 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0926 01:12:27.200390 (+ 195us) server_negotiation.cc:299] Negotiation successful
0926 01:12:27.200470 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
Sep 26 01:12:27 dist-test-slave-7b22 krb5kdc[13685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385146, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260926 01:12:27.201671 13922 sys_catalog.cc:455] T 00000000000000000000000000000000 P 804353922ef94f73b5a375e74d61ba1b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da24713c634743a04ecc86dfba3145" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33013 } } peers { permanent_uuid: "272ffbb71f304dedb633e7eb29e42513" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43779 } } peers { permanent_uuid: "804353922ef94f73b5a375e74d61ba1b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34667 } } }
I20260926 01:12:27.202276 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260926 01:12:27.202553 13933 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 804353922ef94f73b5a375e74d61ba1b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 01:12:27.202590 13933 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 01:12:27.202207 13922 sys_catalog.cc:458] T 00000000000000000000000000000000 P 804353922ef94f73b5a375e74d61ba1b [sys.catalog]: This master's current role is: FOLLOWER
I20260926 01:12:27.204022 13768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:27.196623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34667 (local address 127.0.0.1:51320)
0926 01:12:27.196674 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0926 01:12:27.196683 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 01:12:27.196719 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:12:27.197155 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:12:27.197156 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:12:27.197296 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:12:27.197338 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:12:27.197341 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:12:27.198153 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:12:27.198154 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:12:27.199147 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:12:27.199150 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:12:27.199510 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:12:27.199512 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:12:27.199574 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:12:27.199898 (+ 324us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:12:27.199903 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:12:27.201260 (+ 1357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:12:27.202933 (+ 1673us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:12:27.202934 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:12:27.202935 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:12:27.203153 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:12:27.203352 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:12:27.203352 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:12:27.203353 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:12:27.203414 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:12:27.203717 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:12:27.203718 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:12:27.203766 (+ 48us) client_negotiation.cc:769] Sending connection context
0926 01:12:27.203824 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 01:12:27.203917 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260926 01:12:27.204278 13920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:27.197007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51320 (local address 127.0.98.60:34667)
0926 01:12:27.197053 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 01:12:27.197055 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:12:27.197062 (+ 7us) server_negotiation.cc:407] Connection header received
0926 01:12:27.197080 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:12:27.197081 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:12:27.197089 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:12:27.197169 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:12:27.197404 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:12:27.198065 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:12:27.199212 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:12:27.199414 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:12:27.199498 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:12:27.201355 (+ 1857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:12:27.201357 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:12:27.201357 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:12:27.201364 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:12:27.202846 (+ 1482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:12:27.203227 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:12:27.203229 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:12:27.203229 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:12:27.203285 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:12:27.203476 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:12:27.203477 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:12:27.203477 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:12:27.203627 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:12:27.203689 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 01:12:27.204130 (+ 441us) server_negotiation.cc:299] Negotiation successful
0926 01:12:27.204200 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 01:12:27.202530 13848 master_runner.cc:186] Error getting master registration for 127.0.98.62:33013: OK, instance_id { permanent_uuid: "94da24713c634743a04ecc86dfba3145" instance_seqno: 1790385146851195 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 01:12:27.205013 13848 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 01:12:27.205698 13766 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94da24713c634743a04ecc86dfba3145: Bootstrap starting.
I20260926 01:12:27.206305 13766 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 94da24713c634743a04ecc86dfba3145: Neither blocks nor log segments found. Creating new log.
I20260926 01:12:27.206564 13766 log.cc:824] T 00000000000000000000000000000000 P 94da24713c634743a04ecc86dfba3145: Log is configured to *not* fsync() on all Append() calls
I20260926 01:12:27.207276 13766 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94da24713c634743a04ecc86dfba3145: No bootstrap required, opened a new log
I20260926 01:12:27.208614 13766 raft_consensus.cc:348] T 00000000000000000000000000000000 P 94da24713c634743a04ecc86dfba3145 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da24713c634743a04ecc86dfba3145" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33013 } } peers { permanent_uuid: "272ffbb71f304dedb633e7eb29e42513" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43779 } } peers { permanent_uuid: "804353922ef94f73b5a375e74d61ba1b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34667 } }
I20260926 01:12:27.208720 13766 raft_consensus.cc:374] T 00000000000000000000000000000000 P 94da24713c634743a04ecc86dfba3145 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:12:27.208752 13766 raft_consensus.cc:729] T 00000000000000000000000000000000 P 94da24713c634743a04ecc86dfba3145 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94da24713c634743a04ecc86dfba3145, State: Initialized, Role: FOLLOWER
I20260926 01:12:27.208863 13766 consensus_queue.cc:260] T 00000000000000000000000000000000 P 94da24713c634743a04ecc86dfba3145 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da24713c634743a04ecc86dfba3145" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33013 } } peers { permanent_uuid: "272ffbb71f304dedb633e7eb29e42513" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43779 } } peers { permanent_uuid: "804353922ef94f73b5a375e74d61ba1b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34667 } }
I20260926 01:12:27.209108 13943 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94da24713c634743a04ecc86dfba3145 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da24713c634743a04ecc86dfba3145" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33013 } } peers { permanent_uuid: "272ffbb71f304dedb633e7eb29e42513" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43779 } } peers { permanent_uuid: "804353922ef94f73b5a375e74d61ba1b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34667 } } }
I20260926 01:12:27.209173 13766 sys_catalog.cc:564] T 00000000000000000000000000000000 P 94da24713c634743a04ecc86dfba3145 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:12:27.209183 13943 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94da24713c634743a04ecc86dfba3145 [sys.catalog]: This master's current role is: FOLLOWER
W20260926 01:12:27.210944 13954 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 94da24713c634743a04ecc86dfba3145: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 01:12:27.211002 13954 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 01:12:27.211598 13698 master_runner.cc:186] Error getting master registration for 127.0.98.61:43779: OK, instance_id { permanent_uuid: "272ffbb71f304dedb633e7eb29e42513" instance_seqno: 1790385147008700 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 01:12:27.214350 13698 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 01:12:27.218140 13920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:27.205000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51324 (local address 127.0.98.60:34667)
0926 01:12:27.205030 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0926 01:12:27.205032 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:12:27.210263 (+ 5231us) server_negotiation.cc:407] Connection header received
0926 01:12:27.210356 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:12:27.210358 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:12:27.210368 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:12:27.210396 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:12:27.211086 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:12:27.211786 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:12:27.212760 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:12:27.213027 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:12:27.213069 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:12:27.214544 (+ 1475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:12:27.214545 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:12:27.214545 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:12:27.214552 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:12:27.216081 (+ 1529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:12:27.217226 (+ 1145us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:12:27.217227 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:12:27.217228 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:12:27.217287 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:12:27.217504 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:12:27.217505 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:12:27.217505 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:12:27.217637 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:12:27.217707 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 01:12:27.217971 (+ 264us) server_negotiation.cc:299] Negotiation successful
0926 01:12:27.218047 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260926 01:12:27.219897 13769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:27.203739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33064 (local address 127.0.98.62:33013)
0926 01:12:27.203781 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 01:12:27.203783 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:12:27.210005 (+ 6222us) server_negotiation.cc:407] Connection header received
0926 01:12:27.210057 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:12:27.210058 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:12:27.210069 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:12:27.210129 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:12:27.211097 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:12:27.211730 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:12:27.215060 (+ 3330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:12:27.215339 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:12:27.215381 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:12:27.217213 (+ 1832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:12:27.217214 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:12:27.217215 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:12:27.217222 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:12:27.218696 (+ 1474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:12:27.219072 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:12:27.219074 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:12:27.219074 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:12:27.219133 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:12:27.219325 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:12:27.219326 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:12:27.219327 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:12:27.219466 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:12:27.219537 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 01:12:27.219741 (+ 204us) server_negotiation.cc:299] Negotiation successful
0926 01:12:27.219819 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 01:12:27.222762 13845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:27.204692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59620 (local address 127.0.98.61:43779)
0926 01:12:27.204729 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 01:12:27.204731 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:12:27.210172 (+ 5441us) server_negotiation.cc:407] Connection header received
0926 01:12:27.210198 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:12:27.210199 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:12:27.210210 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:12:27.210253 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:12:27.213290 (+ 3037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:12:27.214023 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:12:27.215739 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:12:27.216031 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:12:27.216073 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:12:27.218795 (+ 2722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:12:27.218796 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:12:27.218797 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:12:27.218804 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:12:27.221545 (+ 2741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:12:27.221955 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:12:27.221956 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:12:27.221956 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:12:27.222025 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:12:27.222231 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:12:27.222233 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:12:27.222233 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:12:27.222375 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:12:27.222493 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 01:12:27.222599 (+ 106us) server_negotiation.cc:299] Negotiation successful
0926 01:12:27.222675 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260926 01:12:27.243467 13841 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34667 } attempt: 3
Sep 26 01:12:27 dist-test-slave-7b22 krb5kdc[13685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385146, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260926 01:12:27.252084 13920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:27.243915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51336 (local address 127.0.98.60:34667)
0926 01:12:27.243959 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 01:12:27.243961 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:12:27.243971 (+ 10us) server_negotiation.cc:407] Connection header received
0926 01:12:27.244053 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:12:27.244054 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:12:27.244063 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:12:27.244120 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:12:27.244587 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:12:27.245415 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:12:27.246509 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:12:27.246810 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:12:27.246956 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:12:27.249247 (+ 2291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:12:27.249248 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:12:27.249249 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:12:27.249257 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:12:27.250919 (+ 1662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:12:27.251304 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:12:27.251305 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:12:27.251305 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:12:27.251364 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:12:27.251560 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:12:27.251560 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:12:27.251561 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:12:27.251695 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:12:27.251759 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 01:12:27.251899 (+ 140us) server_negotiation.cc:299] Negotiation successful
0926 01:12:27.251988 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 01:12:27.252084 13843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:27.243823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34667 (local address 127.0.0.1:51336)
0926 01:12:27.243903 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0926 01:12:27.243917 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 01:12:27.243978 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:12:27.244232 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:12:27.244233 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:12:27.244449 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:12:27.244507 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:12:27.244509 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:12:27.245510 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:12:27.245511 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:12:27.246403 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:12:27.246415 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:12:27.246888 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:12:27.246890 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:12:27.247071 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:12:27.247580 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:12:27.247585 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:12:27.249147 (+ 1562us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:12:27.251006 (+ 1859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:12:27.251008 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:12:27.251008 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:12:27.251231 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:12:27.251441 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:12:27.251442 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:12:27.251442 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:12:27.251503 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:12:27.251753 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:12:27.251754 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:12:27.251803 (+ 49us) client_negotiation.cc:769] Sending connection context
0926 01:12:27.251868 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 01:12:27.251975 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260926 01:12:27.254292 13841 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 272ffbb71f304dedb633e7eb29e42513: Bootstrap starting.
I20260926 01:12:27.255065 13841 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 272ffbb71f304dedb633e7eb29e42513: Neither blocks nor log segments found. Creating new log.
I20260926 01:12:27.255354 13841 log.cc:824] T 00000000000000000000000000000000 P 272ffbb71f304dedb633e7eb29e42513: Log is configured to *not* fsync() on all Append() calls
I20260926 01:12:27.256196 13841 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 272ffbb71f304dedb633e7eb29e42513: No bootstrap required, opened a new log
I20260926 01:12:27.258194 13841 raft_consensus.cc:348] T 00000000000000000000000000000000 P 272ffbb71f304dedb633e7eb29e42513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da24713c634743a04ecc86dfba3145" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33013 } } peers { permanent_uuid: "272ffbb71f304dedb633e7eb29e42513" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43779 } } peers { permanent_uuid: "804353922ef94f73b5a375e74d61ba1b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34667 } }
I20260926 01:12:27.258332 13841 raft_consensus.cc:374] T 00000000000000000000000000000000 P 272ffbb71f304dedb633e7eb29e42513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:12:27.258359 13841 raft_consensus.cc:729] T 00000000000000000000000000000000 P 272ffbb71f304dedb633e7eb29e42513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 272ffbb71f304dedb633e7eb29e42513, State: Initialized, Role: FOLLOWER
I20260926 01:12:27.258472 13841 consensus_queue.cc:260] T 00000000000000000000000000000000 P 272ffbb71f304dedb633e7eb29e42513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da24713c634743a04ecc86dfba3145" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33013 } } peers { permanent_uuid: "272ffbb71f304dedb633e7eb29e42513" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43779 } } peers { permanent_uuid: "804353922ef94f73b5a375e74d61ba1b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34667 } }
I20260926 01:12:27.258797 13841 sys_catalog.cc:564] T 00000000000000000000000000000000 P 272ffbb71f304dedb633e7eb29e42513 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:12:27.258759 13957 sys_catalog.cc:455] T 00000000000000000000000000000000 P 272ffbb71f304dedb633e7eb29e42513 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da24713c634743a04ecc86dfba3145" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33013 } } peers { permanent_uuid: "272ffbb71f304dedb633e7eb29e42513" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43779 } } peers { permanent_uuid: "804353922ef94f73b5a375e74d61ba1b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34667 } } }
I20260926 01:12:27.258940 13957 sys_catalog.cc:458] T 00000000000000000000000000000000 P 272ffbb71f304dedb633e7eb29e42513 [sys.catalog]: This master's current role is: FOLLOWER
W20260926 01:12:27.260646 13968 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 272ffbb71f304dedb633e7eb29e42513: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 01:12:27.260752 13968 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 01:12:27.263196 13773 master_runner.cc:263] No leader master found from master 272ffbb71f304dedb633e7eb29e42513
I20260926 01:12:27.263249 13773 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 01:12:27.332975 13957 raft_consensus.cc:482] T 00000000000000000000000000000000 P 272ffbb71f304dedb633e7eb29e42513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:12:27.333129 13957 raft_consensus.cc:504] T 00000000000000000000000000000000 P 272ffbb71f304dedb633e7eb29e42513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da24713c634743a04ecc86dfba3145" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33013 } } peers { permanent_uuid: "272ffbb71f304dedb633e7eb29e42513" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43779 } } peers { permanent_uuid: "804353922ef94f73b5a375e74d61ba1b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34667 } }
I20260926 01:12:27.333379 13957 leader_election.cc:288] T 00000000000000000000000000000000 P 272ffbb71f304dedb633e7eb29e42513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94da24713c634743a04ecc86dfba3145 (127.0.98.62:33013), 804353922ef94f73b5a375e74d61ba1b (127.0.98.60:34667)
I20260926 01:12:27.333884 13740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "272ffbb71f304dedb633e7eb29e42513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94da24713c634743a04ecc86dfba3145" is_pre_election: true
I20260926 01:12:27.334033 13740 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 94da24713c634743a04ecc86dfba3145 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 272ffbb71f304dedb633e7eb29e42513 in term 0.
I20260926 01:12:27.334354 13791 leader_election.cc:302] T 00000000000000000000000000000000 P 272ffbb71f304dedb633e7eb29e42513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 272ffbb71f304dedb633e7eb29e42513, 94da24713c634743a04ecc86dfba3145; no voters:
I20260926 01:12:27.334535 13957 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 272ffbb71f304dedb633e7eb29e42513 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:12:27.334579 13957 raft_consensus.cc:482] T 00000000000000000000000000000000 P 272ffbb71f304dedb633e7eb29e42513 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:12:27.334597 13957 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 272ffbb71f304dedb633e7eb29e42513 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:12:27.335479 13957 raft_consensus.cc:504] T 00000000000000000000000000000000 P 272ffbb71f304dedb633e7eb29e42513 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da24713c634743a04ecc86dfba3145" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33013 } } peers { permanent_uuid: "272ffbb71f304dedb633e7eb29e42513" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43779 } } peers { permanent_uuid: "804353922ef94f73b5a375e74d61ba1b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34667 } }
I20260926 01:12:27.335599 13957 leader_election.cc:288] T 00000000000000000000000000000000 P 272ffbb71f304dedb633e7eb29e42513 [CANDIDATE]: Term 1 election: Requested vote from peers 94da24713c634743a04ecc86dfba3145 (127.0.98.62:33013), 804353922ef94f73b5a375e74d61ba1b (127.0.98.60:34667)
I20260926 01:12:27.336138 13889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "272ffbb71f304dedb633e7eb29e42513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "804353922ef94f73b5a375e74d61ba1b"
I20260926 01:12:27.336287 13889 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 804353922ef94f73b5a375e74d61ba1b [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:12:27.337137 13889 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 804353922ef94f73b5a375e74d61ba1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 272ffbb71f304dedb633e7eb29e42513 in term 1.
I20260926 01:12:27.337649 13890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "272ffbb71f304dedb633e7eb29e42513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "804353922ef94f73b5a375e74d61ba1b" is_pre_election: true
I20260926 01:12:27.337742 13890 raft_consensus.cc:2362] T 00000000000000000000000000000000 P 804353922ef94f73b5a375e74d61ba1b [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 272ffbb71f304dedb633e7eb29e42513 in term 1. Re-sending same reply.
I20260926 01:12:27.337984 13740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "272ffbb71f304dedb633e7eb29e42513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94da24713c634743a04ecc86dfba3145"
I20260926 01:12:27.338073 13740 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 94da24713c634743a04ecc86dfba3145 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:12:27.338960 13740 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 94da24713c634743a04ecc86dfba3145 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 272ffbb71f304dedb633e7eb29e42513 in term 1.
I20260926 01:12:27.339382 13791 leader_election.cc:302] T 00000000000000000000000000000000 P 272ffbb71f304dedb633e7eb29e42513 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 272ffbb71f304dedb633e7eb29e42513, 94da24713c634743a04ecc86dfba3145; no voters:
I20260926 01:12:27.340334 13957 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 272ffbb71f304dedb633e7eb29e42513 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:12:27.340523 13957 raft_consensus.cc:686] T 00000000000000000000000000000000 P 272ffbb71f304dedb633e7eb29e42513 [term 1 LEADER]: Becoming Leader. State: Replica: 272ffbb71f304dedb633e7eb29e42513, State: Running, Role: LEADER
I20260926 01:12:27.340626 13957 consensus_queue.cc:237] T 00000000000000000000000000000000 P 272ffbb71f304dedb633e7eb29e42513 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da24713c634743a04ecc86dfba3145" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33013 } } peers { permanent_uuid: "272ffbb71f304dedb633e7eb29e42513" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43779 } } peers { permanent_uuid: "804353922ef94f73b5a375e74d61ba1b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34667 } }
I20260926 01:12:27.340993 13957 sys_catalog.cc:455] T 00000000000000000000000000000000 P 272ffbb71f304dedb633e7eb29e42513 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 272ffbb71f304dedb633e7eb29e42513. Latest consensus state: current_term: 1 leader_uuid: "272ffbb71f304dedb633e7eb29e42513" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da24713c634743a04ecc86dfba3145" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33013 } } peers { permanent_uuid: "272ffbb71f304dedb633e7eb29e42513" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43779 } } peers { permanent_uuid: "804353922ef94f73b5a375e74d61ba1b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34667 } } }
I20260926 01:12:27.341059 13957 sys_catalog.cc:458] T 00000000000000000000000000000000 P 272ffbb71f304dedb633e7eb29e42513 [sys.catalog]: This master's current role is: LEADER
I20260926 01:12:27.342137 13972 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:12:27.342376 13972 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:12:27.344825 13890 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 804353922ef94f73b5a375e74d61ba1b [term 1 FOLLOWER]: Refusing update from remote peer 272ffbb71f304dedb633e7eb29e42513: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:12:27.345250 13970 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 272ffbb71f304dedb633e7eb29e42513 [LEADER]: Connected to new peer: Peer: permanent_uuid: "804353922ef94f73b5a375e74d61ba1b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:12:27.346669 13922 sys_catalog.cc:455] T 00000000000000000000000000000000 P 804353922ef94f73b5a375e74d61ba1b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 272ffbb71f304dedb633e7eb29e42513. Latest consensus state: current_term: 1 leader_uuid: "272ffbb71f304dedb633e7eb29e42513" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da24713c634743a04ecc86dfba3145" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33013 } } peers { permanent_uuid: "272ffbb71f304dedb633e7eb29e42513" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43779 } } peers { permanent_uuid: "804353922ef94f73b5a375e74d61ba1b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34667 } } }
I20260926 01:12:27.346755 13922 sys_catalog.cc:458] T 00000000000000000000000000000000 P 804353922ef94f73b5a375e74d61ba1b [sys.catalog]: This master's current role is: FOLLOWER
I20260926 01:12:27.347307 13740 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 94da24713c634743a04ecc86dfba3145 [term 1 FOLLOWER]: Refusing update from remote peer 272ffbb71f304dedb633e7eb29e42513: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:12:27.347602 13970 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 272ffbb71f304dedb633e7eb29e42513 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94da24713c634743a04ecc86dfba3145" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:12:27.348686 13943 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94da24713c634743a04ecc86dfba3145 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 272ffbb71f304dedb633e7eb29e42513. Latest consensus state: current_term: 1 leader_uuid: "272ffbb71f304dedb633e7eb29e42513" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da24713c634743a04ecc86dfba3145" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33013 } } peers { permanent_uuid: "272ffbb71f304dedb633e7eb29e42513" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43779 } } peers { permanent_uuid: "804353922ef94f73b5a375e74d61ba1b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34667 } } }
I20260926 01:12:27.348769 13943 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94da24713c634743a04ecc86dfba3145 [sys.catalog]: This master's current role is: FOLLOWER
W20260926 01:12:27.348935 13719 tablet.cc:2369] T 00000000000000000000000000000000 P 94da24713c634743a04ecc86dfba3145: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 01:12:27.350253 13970 sys_catalog.cc:455] T 00000000000000000000000000000000 P 272ffbb71f304dedb633e7eb29e42513 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "272ffbb71f304dedb633e7eb29e42513" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da24713c634743a04ecc86dfba3145" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33013 } } peers { permanent_uuid: "272ffbb71f304dedb633e7eb29e42513" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43779 } } peers { permanent_uuid: "804353922ef94f73b5a375e74d61ba1b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34667 } } }
I20260926 01:12:27.350332 13970 sys_catalog.cc:458] T 00000000000000000000000000000000 P 272ffbb71f304dedb633e7eb29e42513 [sys.catalog]: This master's current role is: LEADER
I20260926 01:12:27.350811 13943 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94da24713c634743a04ecc86dfba3145 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "272ffbb71f304dedb633e7eb29e42513" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da24713c634743a04ecc86dfba3145" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33013 } } peers { permanent_uuid: "272ffbb71f304dedb633e7eb29e42513" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43779 } } peers { permanent_uuid: "804353922ef94f73b5a375e74d61ba1b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34667 } } }
I20260926 01:12:27.350903 13943 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94da24713c634743a04ecc86dfba3145 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 01:12:27.351143 13973 mvcc.cc:205] Tried to move back new op lower bound from 7333417563520880640 to 7333417563507630080. Current Snapshot: MvccSnapshot[applied={T|T < 7333417563520880640}]
I20260926 01:12:27.351459 13976 mvcc.cc:205] Tried to move back new op lower bound from 7333417563520880640 to 7333417563507630080. Current Snapshot: MvccSnapshot[applied={T|T < 7333417563520880640}]
I20260926 01:12:27.352294 13957 sys_catalog.cc:455] T 00000000000000000000000000000000 P 272ffbb71f304dedb633e7eb29e42513 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "272ffbb71f304dedb633e7eb29e42513" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da24713c634743a04ecc86dfba3145" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33013 } } peers { permanent_uuid: "272ffbb71f304dedb633e7eb29e42513" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43779 } } peers { permanent_uuid: "804353922ef94f73b5a375e74d61ba1b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34667 } } }
I20260926 01:12:27.352366 13957 sys_catalog.cc:458] T 00000000000000000000000000000000 P 272ffbb71f304dedb633e7eb29e42513 [sys.catalog]: This master's current role is: LEADER
I20260926 01:12:27.352751 13922 sys_catalog.cc:455] T 00000000000000000000000000000000 P 804353922ef94f73b5a375e74d61ba1b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "272ffbb71f304dedb633e7eb29e42513" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da24713c634743a04ecc86dfba3145" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33013 } } peers { permanent_uuid: "272ffbb71f304dedb633e7eb29e42513" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43779 } } peers { permanent_uuid: "804353922ef94f73b5a375e74d61ba1b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34667 } } }
I20260926 01:12:27.352829 13922 sys_catalog.cc:458] T 00000000000000000000000000000000 P 804353922ef94f73b5a375e74d61ba1b [sys.catalog]: This master's current role is: FOLLOWER
I20260926 01:12:27.353334 13972 catalog_manager.cc:1347] Generated new cluster ID: 4f2d16b66e6e46acb7695a9f2b41031b
I20260926 01:12:27.353377 13972 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 01:12:27.359160 13974 mvcc.cc:205] Tried to move back new op lower bound from 7333417563520880640 to 7333417563507630080. Current Snapshot: MvccSnapshot[applied={T|T < 7333417563520880640}]
I20260926 01:12:27.362761 13972 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:12:27.363364 13972 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:12:27.368157 13972 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 272ffbb71f304dedb633e7eb29e42513: Generated new TSK 0
I20260926 01:12:27.368340 13972 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 01:12:28.203020 13933 catalog_manager.cc:1259] Loaded cluster ID: 4f2d16b66e6e46acb7695a9f2b41031b
I20260926 01:12:28.203099 13933 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 804353922ef94f73b5a375e74d61ba1b: loading cluster ID for follower catalog manager: success
I20260926 01:12:28.203974 13933 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 804353922ef94f73b5a375e74d61ba1b: acquiring CA information for follower catalog manager: success
I20260926 01:12:28.204274 13933 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 804353922ef94f73b5a375e74d61ba1b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 01:12:28.211423 13954 catalog_manager.cc:1259] Loaded cluster ID: 4f2d16b66e6e46acb7695a9f2b41031b
I20260926 01:12:28.211493 13954 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 94da24713c634743a04ecc86dfba3145: loading cluster ID for follower catalog manager: success
I20260926 01:12:28.212287 13954 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 94da24713c634743a04ecc86dfba3145: acquiring CA information for follower catalog manager: success
I20260926 01:12:28.212579 13954 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 94da24713c634743a04ecc86dfba3145: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 01:12:28.421617 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 13698
I20260926 01:12:28.426208 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 13773
I20260926 01:12:28.430428 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 13848
2026-09-26T01:12:28Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2783 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790384993574762-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 01:12:28 dist-test-slave-7b22 krb5kdc[13987](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 01:12:28 dist-test-slave-7b22 krb5kdc[13987](info): set up 2 sockets
Sep 26 01:12:28 dist-test-slave-7b22 krb5kdc[13987](info): commencing operation
krb5kdc: starting...
W20260926 01:12:29.389670 392 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790384993574762-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790384993574762-392-0/krb5kdc/test-user.keytab.
Sep 26 01:12:29 dist-test-slave-7b22 krb5kdc[13987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385149, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T01:12:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:12:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790384993574762-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790384993574762-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 01:12:29.509070 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790384993574762-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790384993574762-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790384993574762-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37359
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790384993574762-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37359
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:12:29.589704 14000 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:12:29.589986 14000 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:12:29.590021 14000 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:12:29.591265 14000 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:12:29.591312 14000 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:12:29.591331 14000 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:12:29.591346 14000 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:12:29.591362 14000 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:12:29.592924 14000 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790384993574762-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790384993574762-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37359
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790384993574762-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37359
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14000
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790384993574762-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:12:29.593168 14000 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:12:29.593389 14000 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:12:29.598135 14005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:12:29.598151 14008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:12:29.598227 14000 server_base.cc:1034] running on GCE node
W20260926 01:12:29.598158 14006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:12:29.598644 14000 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:12:29.598995 14000 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:12:29.600152 14000 hybrid_clock.cc:648] HybridClock initialized: now 1790385149600129 us; error 33 us; skew 500 ppm
Sep 26 01:12:29 dist-test-slave-7b22 krb5kdc[13987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385149, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:12:29.602670 14000 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 01:12:29.603152 14000 webserver.cc:460] Webserver started at http://127.0.98.62:33335/ using document root <none> and password file <none>
I20260926 01:12:29.603353 14000 fs_manager.cc:359] Metadata directory not provided
I20260926 01:12:29.603392 14000 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:12:29.603513 14000 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:12:29.604560 14000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790384993574762-392-0/minicluster-data/master-0/data/instance:
uuid: "87129ff74bc94bafa19c30c1fa0dd036"
format_stamp: "Formatted at 2026-09-26 01:12:29 on dist-test-slave-7b22"
server_key: "cc3e5985ada4647fe7b5bae19974c04a"
server_key_iv: "7c994bafcbe59b1b7c1c543904f8ad37"
server_key_version: "encryptionkey@0"
I20260926 01:12:29.604882 14000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790384993574762-392-0/minicluster-data/master-0/wal/instance:
uuid: "87129ff74bc94bafa19c30c1fa0dd036"
format_stamp: "Formatted at 2026-09-26 01:12:29 on dist-test-slave-7b22"
server_key: "cc3e5985ada4647fe7b5bae19974c04a"
server_key_iv: "7c994bafcbe59b1b7c1c543904f8ad37"
server_key_version: "encryptionkey@0"
I20260926 01:12:29.606377 14000 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 01:12:29.607089 14015 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:12:29.607242 14000 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 01:12:29.607326 14000 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790384993574762-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790384993574762-392-0/minicluster-data/master-0/wal
uuid: "87129ff74bc94bafa19c30c1fa0dd036"
format_stamp: "Formatted at 2026-09-26 01:12:29 on dist-test-slave-7b22"
server_key: "cc3e5985ada4647fe7b5bae19974c04a"
server_key_iv: "7c994bafcbe59b1b7c1c543904f8ad37"
server_key_version: "encryptionkey@0"
I20260926 01:12:29.607402 14000 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790384993574762-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790384993574762-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790384993574762-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:12:29.619846 14000 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:12:29.622002 14000 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:12:29.622148 14000 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:12:29.625844 14000 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37359
I20260926 01:12:29.625895 14067 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37359 every 8 connection(s)
I20260926 01:12:29.626237 14000 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
I20260926 01:12:29.627106 14068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:12:29.629338 14068 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87129ff74bc94bafa19c30c1fa0dd036: Bootstrap starting.
I20260926 01:12:29.629976 14068 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 87129ff74bc94bafa19c30c1fa0dd036: Neither blocks nor log segments found. Creating new log.
I20260926 01:12:29.630239 14068 log.cc:824] T 00000000000000000000000000000000 P 87129ff74bc94bafa19c30c1fa0dd036: Log is configured to *not* fsync() on all Append() calls
I20260926 01:12:29.631001 14068 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87129ff74bc94bafa19c30c1fa0dd036: No bootstrap required, opened a new log
I20260926 01:12:29.632597 14068 raft_consensus.cc:348] T 00000000000000000000000000000000 P 87129ff74bc94bafa19c30c1fa0dd036 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87129ff74bc94bafa19c30c1fa0dd036" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37359 } }
I20260926 01:12:29.632733 14068 raft_consensus.cc:374] T 00000000000000000000000000000000 P 87129ff74bc94bafa19c30c1fa0dd036 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:12:29.632754 14068 raft_consensus.cc:729] T 00000000000000000000000000000000 P 87129ff74bc94bafa19c30c1fa0dd036 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87129ff74bc94bafa19c30c1fa0dd036, State: Initialized, Role: FOLLOWER
I20260926 01:12:29.632891 14068 consensus_queue.cc:260] T 00000000000000000000000000000000 P 87129ff74bc94bafa19c30c1fa0dd036 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87129ff74bc94bafa19c30c1fa0dd036" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37359 } }
I20260926 01:12:29.632956 14068 raft_consensus.cc:388] T 00000000000000000000000000000000 P 87129ff74bc94bafa19c30c1fa0dd036 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:12:29.632977 14068 raft_consensus.cc:482] T 00000000000000000000000000000000 P 87129ff74bc94bafa19c30c1fa0dd036 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:12:29.633021 14068 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 87129ff74bc94bafa19c30c1fa0dd036 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:12:29.633536 14068 raft_consensus.cc:504] T 00000000000000000000000000000000 P 87129ff74bc94bafa19c30c1fa0dd036 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87129ff74bc94bafa19c30c1fa0dd036" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37359 } }
I20260926 01:12:29.633625 14068 leader_election.cc:302] T 00000000000000000000000000000000 P 87129ff74bc94bafa19c30c1fa0dd036 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 87129ff74bc94bafa19c30c1fa0dd036; no voters:
I20260926 01:12:29.633776 14068 leader_election.cc:288] T 00000000000000000000000000000000 P 87129ff74bc94bafa19c30c1fa0dd036 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:12:29.633812 14071 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 87129ff74bc94bafa19c30c1fa0dd036 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:12:29.633960 14071 raft_consensus.cc:686] T 00000000000000000000000000000000 P 87129ff74bc94bafa19c30c1fa0dd036 [term 1 LEADER]: Becoming Leader. State: Replica: 87129ff74bc94bafa19c30c1fa0dd036, State: Running, Role: LEADER
I20260926 01:12:29.634001 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 14000
I20260926 01:12:29.634065 14068 sys_catalog.cc:564] T 00000000000000000000000000000000 P 87129ff74bc94bafa19c30c1fa0dd036 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:12:29.634096 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790384993574762-392-0/minicluster-data/master-0/wal/instance
I20260926 01:12:29.634083 14071 consensus_queue.cc:237] T 00000000000000000000000000000000 P 87129ff74bc94bafa19c30c1fa0dd036 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87129ff74bc94bafa19c30c1fa0dd036" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37359 } }
I20260926 01:12:29.634291 392 external_mini_cluster.cc:1428] Setting key e61473af878e4e55cd9f90cbb35eea60
I20260926 01:12:29.634564 14072 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87129ff74bc94bafa19c30c1fa0dd036 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "87129ff74bc94bafa19c30c1fa0dd036" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87129ff74bc94bafa19c30c1fa0dd036" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37359 } } }
I20260926 01:12:29.634634 14072 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87129ff74bc94bafa19c30c1fa0dd036 [sys.catalog]: This master's current role is: LEADER
I20260926 01:12:29.634912 14073 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87129ff74bc94bafa19c30c1fa0dd036 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 87129ff74bc94bafa19c30c1fa0dd036. Latest consensus state: current_term: 1 leader_uuid: "87129ff74bc94bafa19c30c1fa0dd036" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87129ff74bc94bafa19c30c1fa0dd036" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37359 } } }
I20260926 01:12:29.635056 14073 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87129ff74bc94bafa19c30c1fa0dd036 [sys.catalog]: This master's current role is: LEADER
I20260926 01:12:29.634925 14081 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:12:29.635730 14081 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 26 01:12:29 dist-test-slave-7b22 krb5kdc[13987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385149, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:12:29.642020 14081 catalog_manager.cc:1347] Generated new cluster ID: da5207c0e1914b31a2eab0f9d8ea66ef
I20260926 01:12:29.642088 14081 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 01:12:29.645624 14083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:29.634896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38730 (local address 127.0.98.62:37359)
0926 01:12:29.635262 (+ 366us) server_negotiation.cc:206] Beginning negotiation
0926 01:12:29.635272 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 01:12:29.635309 (+ 37us) server_negotiation.cc:407] Connection header received
0926 01:12:29.635443 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:12:29.635452 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:12:29.635524 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:12:29.635645 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:12:29.636381 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:12:29.637759 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:12:29.638593 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:12:29.639220 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:12:29.639330 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:12:29.641989 (+ 2659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:12:29.641993 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:12:29.641994 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:12:29.642006 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:12:29.643974 (+ 1968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:12:29.644384 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:12:29.644387 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:12:29.644387 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:12:29.644506 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:12:29.644695 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:12:29.644696 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:12:29.644696 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:12:29.645011 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:12:29.645154 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0926 01:12:29.645260 (+ 106us) server_negotiation.cc:299] Negotiation successful
0926 01:12:29.645408 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":69,"threads_started":1}
I20260926 01:12:29.646672 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 01:12:29.649904 14081 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:12:29.650417 14081 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:12:29.657601 14081 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 87129ff74bc94bafa19c30c1fa0dd036: Generated new TSK 0
I20260926 01:12:29.657779 14081 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 01:12:29.700706 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 14000
2026-09-26T01:12:29Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1269 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 01:12:29 dist-test-slave-7b22 krb5kdc[14095](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 01:12:29 dist-test-slave-7b22 krb5kdc[14095](info): set up 2 sockets
Sep 26 01:12:29 dist-test-slave-7b22 krb5kdc[14095](info): commencing operation
krb5kdc: starting...
W20260926 01:12:30.665689 392 mini_kdc.cc:121] Time spent starting KDC: real 0.948s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/krb5kdc/test-user.keytab.
Sep 26 01:12:30 dist-test-slave-7b22 krb5kdc[14095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385150, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T01:12:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:12:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 01:12:30.784225 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32949
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:32949
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:12:30.863452 14108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:12:30.863674 14108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:12:30.863705 14108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:12:30.864892 14108 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:12:30.864938 14108 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:12:30.864954 14108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:12:30.864970 14108 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:12:30.864985 14108 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:12:30.866524 14108 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:32949
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32949
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14108
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:12:30.866787 14108 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:12:30.867002 14108 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:12:30.871935 14114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:12:30.872020 14113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:12:30.872153 14108 server_base.cc:1034] running on GCE node
W20260926 01:12:30.872020 14116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:12:30.872501 14108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:12:30.872788 14108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:12:30.873929 14108 hybrid_clock.cc:648] HybridClock initialized: now 1790385150873912 us; error 29 us; skew 500 ppm
Sep 26 01:12:30 dist-test-slave-7b22 krb5kdc[14095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385150, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:12:30.876442 14108 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 01:12:30.876896 14108 webserver.cc:460] Webserver started at http://127.0.98.62:46799/ using document root <none> and password file <none>
I20260926 01:12:30.877095 14108 fs_manager.cc:359] Metadata directory not provided
I20260926 01:12:30.877142 14108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:12:30.877267 14108 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:12:30.878353 14108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/master-0/data/instance:
uuid: "396a0b42495f4fd9a266d7c98fa4012f"
format_stamp: "Formatted at 2026-09-26 01:12:30 on dist-test-slave-7b22"
server_key: "34eef3d71352829a61341ec88d1333f2"
server_key_iv: "a3904234ff1334e45601d2cf3db600ba"
server_key_version: "encryptionkey@0"
I20260926 01:12:30.878686 14108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/master-0/wal/instance:
uuid: "396a0b42495f4fd9a266d7c98fa4012f"
format_stamp: "Formatted at 2026-09-26 01:12:30 on dist-test-slave-7b22"
server_key: "34eef3d71352829a61341ec88d1333f2"
server_key_iv: "a3904234ff1334e45601d2cf3db600ba"
server_key_version: "encryptionkey@0"
I20260926 01:12:30.879983 14108 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 01:12:30.880740 14123 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:12:30.881018 14108 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:12:30.881117 14108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/master-0/wal
uuid: "396a0b42495f4fd9a266d7c98fa4012f"
format_stamp: "Formatted at 2026-09-26 01:12:30 on dist-test-slave-7b22"
server_key: "34eef3d71352829a61341ec88d1333f2"
server_key_iv: "a3904234ff1334e45601d2cf3db600ba"
server_key_version: "encryptionkey@0"
I20260926 01:12:30.881186 14108 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:12:30.903924 14108 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:12:30.906016 14108 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:12:30.906162 14108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:12:30.909813 14108 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:32949
I20260926 01:12:30.909895 14175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:32949 every 8 connection(s)
I20260926 01:12:30.910208 14108 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
I20260926 01:12:30.911255 14176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:12:30.913700 14176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 396a0b42495f4fd9a266d7c98fa4012f: Bootstrap starting.
I20260926 01:12:30.914345 14176 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 396a0b42495f4fd9a266d7c98fa4012f: Neither blocks nor log segments found. Creating new log.
I20260926 01:12:30.914603 14176 log.cc:824] T 00000000000000000000000000000000 P 396a0b42495f4fd9a266d7c98fa4012f: Log is configured to *not* fsync() on all Append() calls
I20260926 01:12:30.915421 14176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 396a0b42495f4fd9a266d7c98fa4012f: No bootstrap required, opened a new log
I20260926 01:12:30.917099 14176 raft_consensus.cc:348] T 00000000000000000000000000000000 P 396a0b42495f4fd9a266d7c98fa4012f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "396a0b42495f4fd9a266d7c98fa4012f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32949 } }
I20260926 01:12:30.917250 14176 raft_consensus.cc:374] T 00000000000000000000000000000000 P 396a0b42495f4fd9a266d7c98fa4012f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:12:30.917270 14176 raft_consensus.cc:729] T 00000000000000000000000000000000 P 396a0b42495f4fd9a266d7c98fa4012f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 396a0b42495f4fd9a266d7c98fa4012f, State: Initialized, Role: FOLLOWER
I20260926 01:12:30.917382 14176 consensus_queue.cc:260] T 00000000000000000000000000000000 P 396a0b42495f4fd9a266d7c98fa4012f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "396a0b42495f4fd9a266d7c98fa4012f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32949 } }
I20260926 01:12:30.917462 14176 raft_consensus.cc:388] T 00000000000000000000000000000000 P 396a0b42495f4fd9a266d7c98fa4012f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:12:30.917488 14176 raft_consensus.cc:482] T 00000000000000000000000000000000 P 396a0b42495f4fd9a266d7c98fa4012f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:12:30.917521 14176 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 396a0b42495f4fd9a266d7c98fa4012f [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:12:30.918426 14176 raft_consensus.cc:504] T 00000000000000000000000000000000 P 396a0b42495f4fd9a266d7c98fa4012f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "396a0b42495f4fd9a266d7c98fa4012f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32949 } }
I20260926 01:12:30.918520 14176 leader_election.cc:302] T 00000000000000000000000000000000 P 396a0b42495f4fd9a266d7c98fa4012f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 396a0b42495f4fd9a266d7c98fa4012f; no voters:
I20260926 01:12:30.918663 14176 leader_election.cc:288] T 00000000000000000000000000000000 P 396a0b42495f4fd9a266d7c98fa4012f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:12:30.918737 14179 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 396a0b42495f4fd9a266d7c98fa4012f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:12:30.918896 14176 sys_catalog.cc:564] T 00000000000000000000000000000000 P 396a0b42495f4fd9a266d7c98fa4012f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:12:30.918951 14179 raft_consensus.cc:686] T 00000000000000000000000000000000 P 396a0b42495f4fd9a266d7c98fa4012f [term 1 LEADER]: Becoming Leader. State: Replica: 396a0b42495f4fd9a266d7c98fa4012f, State: Running, Role: LEADER
I20260926 01:12:30.919108 14179 consensus_queue.cc:237] T 00000000000000000000000000000000 P 396a0b42495f4fd9a266d7c98fa4012f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "396a0b42495f4fd9a266d7c98fa4012f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32949 } }
I20260926 01:12:30.919346 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 14108
I20260926 01:12:30.919467 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/master-0/wal/instance
I20260926 01:12:30.919498 14181 sys_catalog.cc:455] T 00000000000000000000000000000000 P 396a0b42495f4fd9a266d7c98fa4012f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 396a0b42495f4fd9a266d7c98fa4012f. Latest consensus state: current_term: 1 leader_uuid: "396a0b42495f4fd9a266d7c98fa4012f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "396a0b42495f4fd9a266d7c98fa4012f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32949 } } }
I20260926 01:12:30.919567 14181 sys_catalog.cc:458] T 00000000000000000000000000000000 P 396a0b42495f4fd9a266d7c98fa4012f [sys.catalog]: This master's current role is: LEADER
I20260926 01:12:30.919674 392 external_mini_cluster.cc:1428] Setting key 1ec4d9fd3978a8b04b1e34e2a73919d8
I20260926 01:12:30.919701 14180 sys_catalog.cc:455] T 00000000000000000000000000000000 P 396a0b42495f4fd9a266d7c98fa4012f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "396a0b42495f4fd9a266d7c98fa4012f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "396a0b42495f4fd9a266d7c98fa4012f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32949 } } }
I20260926 01:12:30.919768 14180 sys_catalog.cc:458] T 00000000000000000000000000000000 P 396a0b42495f4fd9a266d7c98fa4012f [sys.catalog]: This master's current role is: LEADER
I20260926 01:12:30.919778 14188 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:12:30.920321 14188 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:12:30.922142 14188 catalog_manager.cc:1347] Generated new cluster ID: 9c532ce79c0e4442ad110b86d288c165
I20260926 01:12:30.922200 14188 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 01:12:30 dist-test-slave-7b22 krb5kdc[14095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385150, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:12:30.928045 14188 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:12:30.928604 14188 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:12:30.931385 14197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:30.920296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59316 (local address 127.0.98.62:32949)
0926 01:12:30.920578 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0926 01:12:30.920588 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 01:12:30.920624 (+ 36us) server_negotiation.cc:407] Connection header received
0926 01:12:30.920746 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:12:30.920753 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:12:30.920796 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:12:30.920920 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:12:30.921680 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:12:30.923115 (+ 1435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:12:30.924228 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:12:30.924855 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:12:30.924957 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:12:30.927769 (+ 2812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:12:30.927772 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:12:30.927773 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:12:30.927784 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:12:30.929773 (+ 1989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:12:30.930198 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:12:30.930201 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:12:30.930202 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:12:30.930311 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:12:30.930485 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:12:30.930485 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:12:30.930486 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:12:30.930762 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:12:30.930921 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0926 01:12:30.930995 (+ 74us) server_negotiation.cc:299] Negotiation successful
0926 01:12:30.931153 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":53,"threads_started":1}
I20260926 01:12:30.949342 14188 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 396a0b42495f4fd9a266d7c98fa4012f: Generated new TSK 0
I20260926 01:12:30.949550 14188 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 01:12:30.978343 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32949
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:12:31.060001 14202 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:12:31.060218 14202 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:12:31.060242 14202 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:12:31.061365 14202 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:12:31.061398 14202 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:12:31.061424 14202 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 01:12:31.062917 14202 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32949
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14202
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:12:31.063112 14202 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:12:31.063287 14202 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:12:31.068220 14210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:12:31.068248 14208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:12:31.068257 14207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:12:31.068500 14202 server_base.cc:1034] running on GCE node
I20260926 01:12:31.068699 14202 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:12:31.068922 14202 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:12:31.070070 14202 hybrid_clock.cc:648] HybridClock initialized: now 1790385151070057 us; error 30 us; skew 500 ppm
Sep 26 01:12:31 dist-test-slave-7b22 krb5kdc[14095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385151, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:12:31.072675 14202 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 01:12:31.073105 14202 webserver.cc:460] Webserver started at http://127.0.98.1:43983/ using document root <none> and password file <none>
I20260926 01:12:31.073314 14202 fs_manager.cc:359] Metadata directory not provided
I20260926 01:12:31.073360 14202 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:12:31.073544 14202 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:12:31.074626 14202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-0/data/instance:
uuid: "b193498b4e1a4ca091a9e021e7fa1830"
format_stamp: "Formatted at 2026-09-26 01:12:31 on dist-test-slave-7b22"
server_key: "fdd3b728c1189ff7ea60b8bc9bd976b7"
server_key_iv: "c2490517d7a60083a59be57bc4e4ed69"
server_key_version: "encryptionkey@0"
I20260926 01:12:31.074949 14202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-0/wal/instance:
uuid: "b193498b4e1a4ca091a9e021e7fa1830"
format_stamp: "Formatted at 2026-09-26 01:12:31 on dist-test-slave-7b22"
server_key: "fdd3b728c1189ff7ea60b8bc9bd976b7"
server_key_iv: "c2490517d7a60083a59be57bc4e4ed69"
server_key_version: "encryptionkey@0"
I20260926 01:12:31.076216 14202 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 01:12:31.076898 14217 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:12:31.077057 14202 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 01:12:31.077163 14202 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-0/wal
uuid: "b193498b4e1a4ca091a9e021e7fa1830"
format_stamp: "Formatted at 2026-09-26 01:12:31 on dist-test-slave-7b22"
server_key: "fdd3b728c1189ff7ea60b8bc9bd976b7"
server_key_iv: "c2490517d7a60083a59be57bc4e4ed69"
server_key_version: "encryptionkey@0"
I20260926 01:12:31.077225 14202 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:12:31.088783 14202 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:12:31.090962 14202 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:12:31.091104 14202 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:12:31.091322 14202 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:12:31.091629 14202 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:12:31.091663 14202 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:12:31.091693 14202 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:12:31.091717 14202 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:12:31.097214 14202 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40439
I20260926 01:12:31.097297 14330 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40439 every 8 connection(s)
I20260926 01:12:31.097594 14202 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-0/data/info.pb
I20260926 01:12:31.103137 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 14202
I20260926 01:12:31.103245 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-0/wal/instance
I20260926 01:12:31.103440 392 external_mini_cluster.cc:1428] Setting key d7f99d02eb32b5ddc04a9296b1f35c9d
Sep 26 01:12:31 dist-test-slave-7b22 krb5kdc[14095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385151, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:12:31.108060 14197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:31.098356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52865 (local address 127.0.98.62:32949)
0926 01:12:31.098424 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 01:12:31.098426 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:12:31.098615 (+ 189us) server_negotiation.cc:407] Connection header received
0926 01:12:31.099165 (+ 550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:12:31.099167 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:12:31.099178 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:12:31.099226 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:12:31.100221 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:12:31.100739 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:12:31.101728 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:12:31.101988 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:12:31.102047 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:12:31.104834 (+ 2787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:12:31.104835 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:12:31.104836 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:12:31.104846 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:12:31.106572 (+ 1726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:12:31.106969 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:12:31.106971 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:12:31.106971 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:12:31.107044 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:12:31.107278 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:12:31.107279 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:12:31.107279 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:12:31.107433 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:12:31.107509 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 01:12:31.107837 (+ 328us) server_negotiation.cc:299] Negotiation successful
0926 01:12:31.107953 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 01:12:31.108175 14333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:31.098318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32949 (local address 127.0.98.1:52865)
0926 01:12:31.098495 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 01:12:31.098532 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 01:12:31.099059 (+ 527us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:12:31.099309 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:12:31.099313 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:12:31.099900 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:12:31.100118 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:12:31.100123 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:12:31.100825 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:12:31.100826 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:12:31.101635 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:12:31.101637 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:12:31.102064 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:12:31.102065 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:12:31.102185 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:12:31.102787 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:12:31.102797 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:12:31.104720 (+ 1923us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:12:31.106664 (+ 1944us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:12:31.106671 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:12:31.106673 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:12:31.106904 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:12:31.107114 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:12:31.107115 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:12:31.107115 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:12:31.107217 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:12:31.107527 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:12:31.107536 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:12:31.107685 (+ 149us) client_negotiation.cc:769] Sending connection context
0926 01:12:31.107791 (+ 106us) client_negotiation.cc:240] Negotiation successful
0926 01:12:31.107984 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":82,"threads_started":1}
I20260926 01:12:31.108732 14331 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32949
I20260926 01:12:31.108865 14331 heartbeater.cc:464] Registering TS with master...
I20260926 01:12:31.109078 14331 heartbeater.cc:511] Master 127.0.98.62:32949 requested a full tablet report, sending...
I20260926 01:12:31.109649 14140 ts_manager.cc:194] Registered new tserver with Master: b193498b4e1a4ca091a9e021e7fa1830 (127.0.98.1:40439)
I20260926 01:12:31.110340 14140 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52865
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 01:12:31.130679 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32949
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:12:31.210302 14336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:12:31.210522 14336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:12:31.210546 14336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:12:31.211711 14336 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:12:31.211756 14336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:12:31.211783 14336 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 01:12:31.213254 14336 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32949
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14336
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:12:31.213480 14336 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:12:31.213696 14336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:12:31.218665 14344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:12:31.218669 14342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:12:31.218669 14341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:12:31.219002 14336 server_base.cc:1034] running on GCE node
I20260926 01:12:31.219218 14336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:12:31.219501 14336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:12:31.220652 14336 hybrid_clock.cc:648] HybridClock initialized: now 1790385151220641 us; error 43 us; skew 500 ppm
Sep 26 01:12:31 dist-test-slave-7b22 krb5kdc[14095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385151, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:12:31.223210 14336 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 01:12:31.223662 14336 webserver.cc:460] Webserver started at http://127.0.98.2:37981/ using document root <none> and password file <none>
I20260926 01:12:31.223870 14336 fs_manager.cc:359] Metadata directory not provided
I20260926 01:12:31.223919 14336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:12:31.224040 14336 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:12:31.225114 14336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-1/data/instance:
uuid: "cb9887afb40444c2b400da5aeff93486"
format_stamp: "Formatted at 2026-09-26 01:12:31 on dist-test-slave-7b22"
server_key: "9349ff8e45cff9d62e0eb3b58ca443e3"
server_key_iv: "73f73772a4aeec2ab5af356f748f85df"
server_key_version: "encryptionkey@0"
I20260926 01:12:31.225440 14336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-1/wal/instance:
uuid: "cb9887afb40444c2b400da5aeff93486"
format_stamp: "Formatted at 2026-09-26 01:12:31 on dist-test-slave-7b22"
server_key: "9349ff8e45cff9d62e0eb3b58ca443e3"
server_key_iv: "73f73772a4aeec2ab5af356f748f85df"
server_key_version: "encryptionkey@0"
I20260926 01:12:31.226735 14336 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 01:12:31.227516 14351 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:12:31.227692 14336 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 01:12:31.227777 14336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-1/wal
uuid: "cb9887afb40444c2b400da5aeff93486"
format_stamp: "Formatted at 2026-09-26 01:12:31 on dist-test-slave-7b22"
server_key: "9349ff8e45cff9d62e0eb3b58ca443e3"
server_key_iv: "73f73772a4aeec2ab5af356f748f85df"
server_key_version: "encryptionkey@0"
I20260926 01:12:31.227834 14336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:12:31.254592 14336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:12:31.257046 14336 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:12:31.257227 14336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:12:31.257460 14336 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:12:31.257786 14336 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:12:31.257853 14336 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:12:31.257892 14336 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:12:31.257916 14336 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:12:31.264760 14336 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40171
I20260926 01:12:31.264822 14464 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40171 every 8 connection(s)
I20260926 01:12:31.265166 14336 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-1/data/info.pb
I20260926 01:12:31.265789 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 14336
I20260926 01:12:31.265923 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-1/wal/instance
I20260926 01:12:31.266125 392 external_mini_cluster.cc:1428] Setting key b963d5a46fe5d3fc0424999fa68e69c9
Sep 26 01:12:31 dist-test-slave-7b22 krb5kdc[14095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385151, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260926 01:12:31.275871 14467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:31.265908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32949 (local address 127.0.98.2:54147)
0926 01:12:31.266115 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0926 01:12:31.266168 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 01:12:31.266791 (+ 623us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:12:31.267040 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:12:31.267045 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:12:31.267628 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:12:31.267839 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:12:31.267844 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:12:31.268791 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:12:31.268792 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:12:31.269659 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:12:31.269662 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:12:31.270255 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:12:31.270256 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:12:31.270370 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:12:31.270888 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:12:31.270896 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:12:31.272234 (+ 1338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:12:31.274320 (+ 2086us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:12:31.274329 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:12:31.274332 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:12:31.274620 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:12:31.274861 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:12:31.274861 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:12:31.274862 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:12:31.274987 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:12:31.275282 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:12:31.275289 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:12:31.275434 (+ 145us) client_negotiation.cc:769] Sending connection context
0926 01:12:31.275541 (+ 107us) client_negotiation.cc:240] Negotiation successful
0926 01:12:31.275691 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":84,"threads_started":1}
I20260926 01:12:31.276101 14197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:31.265952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54147 (local address 127.0.98.62:32949)
0926 01:12:31.266021 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0926 01:12:31.266023 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:12:31.266287 (+ 264us) server_negotiation.cc:407] Connection header received
0926 01:12:31.266895 (+ 608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:12:31.266896 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:12:31.266907 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:12:31.266959 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:12:31.267935 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:12:31.268690 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:12:31.269753 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:12:31.270171 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:12:31.270251 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:12:31.272336 (+ 2085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:12:31.272338 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:12:31.272338 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:12:31.272348 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:12:31.274232 (+ 1884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:12:31.274720 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:12:31.274721 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:12:31.274721 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:12:31.274787 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:12:31.275058 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:12:31.275058 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:12:31.275059 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:12:31.275190 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:12:31.275250 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 01:12:31.275813 (+ 563us) server_negotiation.cc:299] Negotiation successful
0926 01:12:31.276010 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260926 01:12:31.276634 14465 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32949
I20260926 01:12:31.276743 14465 heartbeater.cc:464] Registering TS with master...
I20260926 01:12:31.276927 14465 heartbeater.cc:511] Master 127.0.98.62:32949 requested a full tablet report, sending...
I20260926 01:12:31.277323 14140 ts_manager.cc:194] Registered new tserver with Master: cb9887afb40444c2b400da5aeff93486 (127.0.98.2:40171)
I20260926 01:12:31.277717 14140 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:54147
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 01:12:31.292219 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32949
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:12:31.370847 14470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:12:31.371081 14470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:12:31.371105 14470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:12:31.372356 14470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:12:31.372414 14470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:12:31.372443 14470 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 01:12:31.374235 14470 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32949
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14470
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:12:31.374529 14470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:12:31.374769 14470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:12:31.379978 14478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:12:31.380021 14476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:12:31.380108 14475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:12:31.380214 14470 server_base.cc:1034] running on GCE node
I20260926 01:12:31.380816 14470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:12:31.381076 14470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:12:31.382249 14470 hybrid_clock.cc:648] HybridClock initialized: now 1790385151382183 us; error 81 us; skew 500 ppm
Sep 26 01:12:31 dist-test-slave-7b22 krb5kdc[14095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385151, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:12:31.385326 14470 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 01:12:31.385912 14470 webserver.cc:460] Webserver started at http://127.0.98.3:44729/ using document root <none> and password file <none>
I20260926 01:12:31.386127 14470 fs_manager.cc:359] Metadata directory not provided
I20260926 01:12:31.386175 14470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:12:31.386305 14470 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:12:31.387414 14470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-2/data/instance:
uuid: "5e3d9f9c88ff4617b5314f95cb3b5d47"
format_stamp: "Formatted at 2026-09-26 01:12:31 on dist-test-slave-7b22"
server_key: "8bfee5695dbe2c9914163b20a004de48"
server_key_iv: "1c2a4134373cdf7cf9808899feaa641f"
server_key_version: "encryptionkey@0"
I20260926 01:12:31.387763 14470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-2/wal/instance:
uuid: "5e3d9f9c88ff4617b5314f95cb3b5d47"
format_stamp: "Formatted at 2026-09-26 01:12:31 on dist-test-slave-7b22"
server_key: "8bfee5695dbe2c9914163b20a004de48"
server_key_iv: "1c2a4134373cdf7cf9808899feaa641f"
server_key_version: "encryptionkey@0"
I20260926 01:12:31.389114 14470 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 01:12:31.389984 14485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:12:31.390168 14470 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 01:12:31.390267 14470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-2/wal
uuid: "5e3d9f9c88ff4617b5314f95cb3b5d47"
format_stamp: "Formatted at 2026-09-26 01:12:31 on dist-test-slave-7b22"
server_key: "8bfee5695dbe2c9914163b20a004de48"
server_key_iv: "1c2a4134373cdf7cf9808899feaa641f"
server_key_version: "encryptionkey@0"
I20260926 01:12:31.390337 14470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:12:31.405058 14470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:12:31.407131 14470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:12:31.407269 14470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:12:31.407493 14470 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:12:31.407799 14470 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:12:31.407831 14470 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:12:31.407861 14470 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:12:31.407884 14470 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:12:31.413225 14470 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45841
I20260926 01:12:31.413290 14598 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45841 every 8 connection(s)
I20260926 01:12:31.413571 14470 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-2/data/info.pb
I20260926 01:12:31.417105 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 14470
I20260926 01:12:31.417220 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-2/wal/instance
I20260926 01:12:31.417423 392 external_mini_cluster.cc:1428] Setting key a1d4cf43779406b33e3c110a8a2ef462
Sep 26 01:12:31 dist-test-slave-7b22 krb5kdc[14095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385151, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:12:31.422787 14197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:31.414293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55699 (local address 127.0.98.62:32949)
0926 01:12:31.414359 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 01:12:31.414361 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:12:31.414531 (+ 170us) server_negotiation.cc:407] Connection header received
0926 01:12:31.415066 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:12:31.415068 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:12:31.415078 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:12:31.415131 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:12:31.416080 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:12:31.416569 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:12:31.417809 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:12:31.418070 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:12:31.418129 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:12:31.420263 (+ 2134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:12:31.420264 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:12:31.420265 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:12:31.420271 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:12:31.421530 (+ 1259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:12:31.421956 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:12:31.421957 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:12:31.421958 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:12:31.422010 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:12:31.422228 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:12:31.422228 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:12:31.422229 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:12:31.422345 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:12:31.422399 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 01:12:31.422650 (+ 251us) server_negotiation.cc:299] Negotiation successful
0926 01:12:31.422718 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 01:12:31.422950 14601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:31.414245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32949 (local address 127.0.98.3:55699)
0926 01:12:31.414417 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0926 01:12:31.414455 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 01:12:31.414983 (+ 528us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:12:31.415191 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:12:31.415195 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:12:31.415795 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:12:31.416002 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:12:31.416007 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:12:31.416640 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:12:31.416641 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:12:31.417727 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:12:31.417730 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:12:31.418146 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:12:31.418147 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:12:31.418287 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:12:31.418743 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:12:31.418751 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:12:31.420186 (+ 1435us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:12:31.421607 (+ 1421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:12:31.421613 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:12:31.421614 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:12:31.421842 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:12:31.422065 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:12:31.422066 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:12:31.422066 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:12:31.422163 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:12:31.422397 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:12:31.422401 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:12:31.422515 (+ 114us) client_negotiation.cc:769] Sending connection context
0926 01:12:31.422593 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 01:12:31.422712 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":82,"threads_started":1}
I20260926 01:12:31.423439 14599 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32949
I20260926 01:12:31.423532 14599 heartbeater.cc:464] Registering TS with master...
I20260926 01:12:31.423676 14599 heartbeater.cc:511] Master 127.0.98.62:32949 requested a full tablet report, sending...
I20260926 01:12:31.424055 14140 ts_manager.cc:194] Registered new tserver with Master: 5e3d9f9c88ff4617b5314f95cb3b5d47 (127.0.98.3:45841)
I20260926 01:12:31.424405 14140 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:55699
I20260926 01:12:31.428958 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 01:12:31.435818 14197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:31.430296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59326 (local address 127.0.98.62:32949)
0926 01:12:31.430371 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0926 01:12:31.430373 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:12:31.430465 (+ 92us) server_negotiation.cc:407] Connection header received
0926 01:12:31.430530 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:12:31.430531 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:12:31.430541 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:12:31.430594 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:12:31.431146 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:12:31.431681 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:12:31.432373 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:12:31.432565 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:12:31.432611 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:12:31.433713 (+ 1102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:12:31.433714 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:12:31.433714 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:12:31.433721 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:12:31.434929 (+ 1208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:12:31.435239 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:12:31.435240 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:12:31.435240 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:12:31.435288 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:12:31.435419 (+ 131us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:12:31.435420 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:12:31.435420 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:12:31.435554 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:12:31.435623 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 01:12:31.435695 (+ 72us) server_negotiation.cc:299] Negotiation successful
0926 01:12:31.435751 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 01:12:31.437381 14140 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59326:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:12:31.439229 14140 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:12:31.445327 14610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:31.441515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45841 (local address 127.0.0.1:38434)
0926 01:12:31.441862 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0926 01:12:31.441871 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 01:12:31.441926 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:12:31.442178 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:12:31.442179 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:12:31.442181 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:12:31.442217 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:12:31.442218 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:12:31.443414 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:12:31.443415 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:12:31.444169 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:12:31.444171 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:12:31.445010 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:12:31.445011 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:12:31.445088 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:12:31.445102 (+ 14us) client_negotiation.cc:769] Sending connection context
0926 01:12:31.445175 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 01:12:31.445229 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":31,"threads_started":1}
I20260926 01:12:31.445441 14612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:31.441672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38434 (local address 127.0.98.3:45841)
0926 01:12:31.441870 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 01:12:31.441874 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 01:12:31.441915 (+ 41us) server_negotiation.cc:407] Connection header received
0926 01:12:31.442012 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:12:31.442015 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:12:31.442031 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:12:31.442100 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:12:31.442287 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:12:31.443326 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:12:31.444252 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:12:31.444926 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:12:31.444999 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:12:31.445057 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 01:12:31.445232 (+ 175us) server_negotiation.cc:299] Negotiation successful
0926 01:12:31.445277 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":89,"threads_started":1}
I20260926 01:12:31.445649 14607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:31.441313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40171 (local address 127.0.0.1:33088)
0926 01:12:31.441496 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0926 01:12:31.441549 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 01:12:31.441618 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:12:31.441850 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:12:31.441853 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:12:31.441855 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:12:31.441983 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:12:31.441985 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:12:31.443327 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:12:31.443327 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:12:31.444412 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:12:31.444414 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:12:31.445332 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:12:31.445333 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:12:31.445454 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:12:31.445461 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 01:12:31.445528 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 01:12:31.445579 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":7,"thread_start_us":55,"threads_started":1}
I20260926 01:12:31.445806 14611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:31.441484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34428 (local address 127.0.98.1:40439)
0926 01:12:31.442074 (+ 590us) server_negotiation.cc:206] Beginning negotiation
0926 01:12:31.442078 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 01:12:31.442087 (+ 9us) server_negotiation.cc:407] Connection header received
0926 01:12:31.442123 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:12:31.442125 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:12:31.442138 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:12:31.442341 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:12:31.442396 (+ 55us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:12:31.443073 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:12:31.444723 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:12:31.445336 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:12:31.445396 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:12:31.445464 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 01:12:31.445685 (+ 221us) server_negotiation.cc:299] Negotiation successful
0926 01:12:31.445727 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":550,"thread_start_us":89,"threads_started":1}
I20260926 01:12:31.445744 14608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:31.441338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40439 (local address 127.0.0.1:34428)
0926 01:12:31.441654 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0926 01:12:31.441664 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 01:12:31.441705 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:12:31.442230 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:12:31.442231 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:12:31.442234 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:12:31.442281 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:12:31.442283 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:12:31.443167 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:12:31.443168 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:12:31.444654 (+ 1486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:12:31.444656 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:12:31.445496 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:12:31.445497 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:12:31.445554 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:12:31.445557 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 01:12:31.445598 (+ 41us) client_negotiation.cc:240] Negotiation successful
0926 01:12:31.445637 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"mutex_wait_us":30,"thread_start_us":74,"threads_started":1}
I20260926 01:12:31.446470 14533 tablet_service.cc:1467] Processing CreateTablet for tablet 17448b80435245e49447803dfc76a4cd (DEFAULT_TABLE table=test-table [id=3e310cb802d84d10a4e406f91ccbfec8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:12:31.446671 14265 tablet_service.cc:1467] Processing CreateTablet for tablet 17448b80435245e49447803dfc76a4cd (DEFAULT_TABLE table=test-table [id=3e310cb802d84d10a4e406f91ccbfec8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:12:31.446794 14533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17448b80435245e49447803dfc76a4cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:12:31.446950 14265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17448b80435245e49447803dfc76a4cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:12:31.449314 14613 tablet_bootstrap.cc:492] T 17448b80435245e49447803dfc76a4cd P 5e3d9f9c88ff4617b5314f95cb3b5d47: Bootstrap starting.
I20260926 01:12:31.449540 14609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:31.441415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33088 (local address 127.0.98.2:40171)
0926 01:12:31.441605 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 01:12:31.441610 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 01:12:31.441622 (+ 12us) server_negotiation.cc:407] Connection header received
0926 01:12:31.441700 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:12:31.441703 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:12:31.441718 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:12:31.441807 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:12:31.442077 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:12:31.443232 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:12:31.444500 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:12:31.445175 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:12:31.445238 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:12:31.445292 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 01:12:31.445880 (+ 588us) server_negotiation.cc:299] Negotiation successful
0926 01:12:31.445930 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":76,"threads_started":1}
I20260926 01:12:31.449628 14614 tablet_bootstrap.cc:492] T 17448b80435245e49447803dfc76a4cd P b193498b4e1a4ca091a9e021e7fa1830: Bootstrap starting.
I20260926 01:12:31.450052 14613 tablet_bootstrap.cc:654] T 17448b80435245e49447803dfc76a4cd P 5e3d9f9c88ff4617b5314f95cb3b5d47: Neither blocks nor log segments found. Creating new log.
I20260926 01:12:31.450217 14399 tablet_service.cc:1467] Processing CreateTablet for tablet 17448b80435245e49447803dfc76a4cd (DEFAULT_TABLE table=test-table [id=3e310cb802d84d10a4e406f91ccbfec8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:12:31.450481 14613 log.cc:824] T 17448b80435245e49447803dfc76a4cd P 5e3d9f9c88ff4617b5314f95cb3b5d47: Log is configured to *not* fsync() on all Append() calls
I20260926 01:12:31.450516 14399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17448b80435245e49447803dfc76a4cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:12:31.451149 14613 tablet_bootstrap.cc:492] T 17448b80435245e49447803dfc76a4cd P 5e3d9f9c88ff4617b5314f95cb3b5d47: No bootstrap required, opened a new log
I20260926 01:12:31.451202 14613 ts_tablet_manager.cc:1397] T 17448b80435245e49447803dfc76a4cd P 5e3d9f9c88ff4617b5314f95cb3b5d47: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 01:12:31.452522 14613 raft_consensus.cc:348] T 17448b80435245e49447803dfc76a4cd P 5e3d9f9c88ff4617b5314f95cb3b5d47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb9887afb40444c2b400da5aeff93486" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40171 } } peers { permanent_uuid: "b193498b4e1a4ca091a9e021e7fa1830" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40439 } } peers { permanent_uuid: "5e3d9f9c88ff4617b5314f95cb3b5d47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45841 } }
I20260926 01:12:31.452730 14613 raft_consensus.cc:374] T 17448b80435245e49447803dfc76a4cd P 5e3d9f9c88ff4617b5314f95cb3b5d47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:12:31.452787 14613 raft_consensus.cc:729] T 17448b80435245e49447803dfc76a4cd P 5e3d9f9c88ff4617b5314f95cb3b5d47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e3d9f9c88ff4617b5314f95cb3b5d47, State: Initialized, Role: FOLLOWER
I20260926 01:12:31.452942 14613 consensus_queue.cc:260] T 17448b80435245e49447803dfc76a4cd P 5e3d9f9c88ff4617b5314f95cb3b5d47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb9887afb40444c2b400da5aeff93486" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40171 } } peers { permanent_uuid: "b193498b4e1a4ca091a9e021e7fa1830" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40439 } } peers { permanent_uuid: "5e3d9f9c88ff4617b5314f95cb3b5d47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45841 } }
I20260926 01:12:31.453222 14614 tablet_bootstrap.cc:654] T 17448b80435245e49447803dfc76a4cd P b193498b4e1a4ca091a9e021e7fa1830: Neither blocks nor log segments found. Creating new log.
I20260926 01:12:31.453236 14616 tablet_bootstrap.cc:492] T 17448b80435245e49447803dfc76a4cd P cb9887afb40444c2b400da5aeff93486: Bootstrap starting.
I20260926 01:12:31.453354 14613 ts_tablet_manager.cc:1428] T 17448b80435245e49447803dfc76a4cd P 5e3d9f9c88ff4617b5314f95cb3b5d47: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 01:12:31.453411 14599 heartbeater.cc:503] Master 127.0.98.62:32949 was elected leader, sending a full tablet report...
I20260926 01:12:31.453806 14614 log.cc:824] T 17448b80435245e49447803dfc76a4cd P b193498b4e1a4ca091a9e021e7fa1830: Log is configured to *not* fsync() on all Append() calls
I20260926 01:12:31.453948 14616 tablet_bootstrap.cc:654] T 17448b80435245e49447803dfc76a4cd P cb9887afb40444c2b400da5aeff93486: Neither blocks nor log segments found. Creating new log.
I20260926 01:12:31.454460 14616 log.cc:824] T 17448b80435245e49447803dfc76a4cd P cb9887afb40444c2b400da5aeff93486: Log is configured to *not* fsync() on all Append() calls
I20260926 01:12:31.454627 14614 tablet_bootstrap.cc:492] T 17448b80435245e49447803dfc76a4cd P b193498b4e1a4ca091a9e021e7fa1830: No bootstrap required, opened a new log
I20260926 01:12:31.454730 14614 ts_tablet_manager.cc:1397] T 17448b80435245e49447803dfc76a4cd P b193498b4e1a4ca091a9e021e7fa1830: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260926 01:12:31.455152 14616 tablet_bootstrap.cc:492] T 17448b80435245e49447803dfc76a4cd P cb9887afb40444c2b400da5aeff93486: No bootstrap required, opened a new log
I20260926 01:12:31.455210 14616 ts_tablet_manager.cc:1397] T 17448b80435245e49447803dfc76a4cd P cb9887afb40444c2b400da5aeff93486: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 01:12:31.456229 14614 raft_consensus.cc:348] T 17448b80435245e49447803dfc76a4cd P b193498b4e1a4ca091a9e021e7fa1830 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb9887afb40444c2b400da5aeff93486" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40171 } } peers { permanent_uuid: "b193498b4e1a4ca091a9e021e7fa1830" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40439 } } peers { permanent_uuid: "5e3d9f9c88ff4617b5314f95cb3b5d47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45841 } }
I20260926 01:12:31.456374 14614 raft_consensus.cc:374] T 17448b80435245e49447803dfc76a4cd P b193498b4e1a4ca091a9e021e7fa1830 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:12:31.456396 14614 raft_consensus.cc:729] T 17448b80435245e49447803dfc76a4cd P b193498b4e1a4ca091a9e021e7fa1830 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b193498b4e1a4ca091a9e021e7fa1830, State: Initialized, Role: FOLLOWER
I20260926 01:12:31.456508 14614 consensus_queue.cc:260] T 17448b80435245e49447803dfc76a4cd P b193498b4e1a4ca091a9e021e7fa1830 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb9887afb40444c2b400da5aeff93486" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40171 } } peers { permanent_uuid: "b193498b4e1a4ca091a9e021e7fa1830" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40439 } } peers { permanent_uuid: "5e3d9f9c88ff4617b5314f95cb3b5d47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45841 } }
I20260926 01:12:31.456771 14614 ts_tablet_manager.cc:1428] T 17448b80435245e49447803dfc76a4cd P b193498b4e1a4ca091a9e021e7fa1830: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 01:12:31.456702 14616 raft_consensus.cc:348] T 17448b80435245e49447803dfc76a4cd P cb9887afb40444c2b400da5aeff93486 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb9887afb40444c2b400da5aeff93486" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40171 } } peers { permanent_uuid: "b193498b4e1a4ca091a9e021e7fa1830" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40439 } } peers { permanent_uuid: "5e3d9f9c88ff4617b5314f95cb3b5d47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45841 } }
I20260926 01:12:31.456808 14331 heartbeater.cc:503] Master 127.0.98.62:32949 was elected leader, sending a full tablet report...
I20260926 01:12:31.456887 14616 raft_consensus.cc:374] T 17448b80435245e49447803dfc76a4cd P cb9887afb40444c2b400da5aeff93486 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:12:31.456938 14616 raft_consensus.cc:729] T 17448b80435245e49447803dfc76a4cd P cb9887afb40444c2b400da5aeff93486 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb9887afb40444c2b400da5aeff93486, State: Initialized, Role: FOLLOWER
I20260926 01:12:31.457047 14616 consensus_queue.cc:260] T 17448b80435245e49447803dfc76a4cd P cb9887afb40444c2b400da5aeff93486 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb9887afb40444c2b400da5aeff93486" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40171 } } peers { permanent_uuid: "b193498b4e1a4ca091a9e021e7fa1830" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40439 } } peers { permanent_uuid: "5e3d9f9c88ff4617b5314f95cb3b5d47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45841 } }
I20260926 01:12:31.457335 14616 ts_tablet_manager.cc:1428] T 17448b80435245e49447803dfc76a4cd P cb9887afb40444c2b400da5aeff93486: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 01:12:31.457485 14465 heartbeater.cc:503] Master 127.0.98.62:32949 was elected leader, sending a full tablet report...
I20260926 01:12:31.499390 14617 raft_consensus.cc:482] T 17448b80435245e49447803dfc76a4cd P 5e3d9f9c88ff4617b5314f95cb3b5d47 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:12:31.499537 14617 raft_consensus.cc:504] T 17448b80435245e49447803dfc76a4cd P 5e3d9f9c88ff4617b5314f95cb3b5d47 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb9887afb40444c2b400da5aeff93486" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40171 } } peers { permanent_uuid: "b193498b4e1a4ca091a9e021e7fa1830" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40439 } } peers { permanent_uuid: "5e3d9f9c88ff4617b5314f95cb3b5d47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45841 } }
I20260926 01:12:31.499913 14617 leader_election.cc:288] T 17448b80435245e49447803dfc76a4cd P 5e3d9f9c88ff4617b5314f95cb3b5d47 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cb9887afb40444c2b400da5aeff93486 (127.0.98.2:40171), b193498b4e1a4ca091a9e021e7fa1830 (127.0.98.1:40439)
I20260926 01:12:31.503275 14609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:31.500140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47771 (local address 127.0.98.2:40171)
0926 01:12:31.500197 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 01:12:31.500198 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 01:12:31.500207 (+ 9us) server_negotiation.cc:407] Connection header received
0926 01:12:31.500364 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:12:31.500365 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:12:31.500382 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:12:31.500455 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:12:31.500653 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:12:31.501254 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:12:31.502392 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:12:31.502971 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:12:31.503023 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:12:31.503032 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 01:12:31.503187 (+ 155us) server_negotiation.cc:299] Negotiation successful
0926 01:12:31.503219 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 01:12:31.503295 14601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:31.500052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40171 (local address 127.0.98.3:47771)
0926 01:12:31.500092 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0926 01:12:31.500103 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 01:12:31.500142 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:12:31.500485 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:12:31.500487 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:12:31.500489 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:12:31.500565 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:12:31.500567 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:12:31.501342 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:12:31.501343 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:12:31.502303 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:12:31.502306 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:12:31.503039 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:12:31.503041 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:12:31.503119 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:12:31.503123 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 01:12:31.503178 (+ 55us) client_negotiation.cc:240] Negotiation successful
0926 01:12:31.503219 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19}
I20260926 01:12:31.503669 14419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17448b80435245e49447803dfc76a4cd" candidate_uuid: "5e3d9f9c88ff4617b5314f95cb3b5d47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb9887afb40444c2b400da5aeff93486" is_pre_election: true
I20260926 01:12:31.503744 14622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:31.500052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40439 (local address 127.0.98.3:50527)
0926 01:12:31.500512 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0926 01:12:31.500522 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 01:12:31.500593 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:12:31.500715 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:12:31.500716 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:12:31.500717 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:12:31.500797 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:12:31.500799 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:12:31.501861 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:12:31.501863 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:12:31.502812 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:12:31.502815 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:12:31.503511 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:12:31.503512 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:12:31.503583 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:12:31.503588 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 01:12:31.503632 (+ 44us) client_negotiation.cc:240] Negotiation successful
0926 01:12:31.503669 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"mutex_wait_us":258,"thread_start_us":109,"threads_started":1}
I20260926 01:12:31.503756 14611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:31.500141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50527 (local address 127.0.98.1:40439)
0926 01:12:31.500273 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0926 01:12:31.500275 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:12:31.500603 (+ 328us) server_negotiation.cc:407] Connection header received
0926 01:12:31.500655 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:12:31.500656 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:12:31.500666 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:12:31.500705 (+ 39us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:12:31.501090 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:12:31.501724 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:12:31.502904 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:12:31.503438 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:12:31.503497 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:12:31.503507 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 01:12:31.503653 (+ 146us) server_negotiation.cc:299] Negotiation successful
0926 01:12:31.503698 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260926 01:12:31.503821 14419 raft_consensus.cc:2454] T 17448b80435245e49447803dfc76a4cd P cb9887afb40444c2b400da5aeff93486 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e3d9f9c88ff4617b5314f95cb3b5d47 in term 0.
I20260926 01:12:31.504093 14488 leader_election.cc:302] T 17448b80435245e49447803dfc76a4cd P 5e3d9f9c88ff4617b5314f95cb3b5d47 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e3d9f9c88ff4617b5314f95cb3b5d47, cb9887afb40444c2b400da5aeff93486; no voters:
I20260926 01:12:31.504114 14285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17448b80435245e49447803dfc76a4cd" candidate_uuid: "5e3d9f9c88ff4617b5314f95cb3b5d47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b193498b4e1a4ca091a9e021e7fa1830" is_pre_election: true
I20260926 01:12:31.504247 14285 raft_consensus.cc:2454] T 17448b80435245e49447803dfc76a4cd P b193498b4e1a4ca091a9e021e7fa1830 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e3d9f9c88ff4617b5314f95cb3b5d47 in term 0.
I20260926 01:12:31.504225 14617 raft_consensus.cc:2781] T 17448b80435245e49447803dfc76a4cd P 5e3d9f9c88ff4617b5314f95cb3b5d47 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:12:31.504289 14617 raft_consensus.cc:482] T 17448b80435245e49447803dfc76a4cd P 5e3d9f9c88ff4617b5314f95cb3b5d47 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:12:31.504320 14617 raft_consensus.cc:3037] T 17448b80435245e49447803dfc76a4cd P 5e3d9f9c88ff4617b5314f95cb3b5d47 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:12:31.505013 14617 raft_consensus.cc:504] T 17448b80435245e49447803dfc76a4cd P 5e3d9f9c88ff4617b5314f95cb3b5d47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb9887afb40444c2b400da5aeff93486" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40171 } } peers { permanent_uuid: "b193498b4e1a4ca091a9e021e7fa1830" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40439 } } peers { permanent_uuid: "5e3d9f9c88ff4617b5314f95cb3b5d47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45841 } }
I20260926 01:12:31.505129 14617 leader_election.cc:288] T 17448b80435245e49447803dfc76a4cd P 5e3d9f9c88ff4617b5314f95cb3b5d47 [CANDIDATE]: Term 1 election: Requested vote from peers cb9887afb40444c2b400da5aeff93486 (127.0.98.2:40171), b193498b4e1a4ca091a9e021e7fa1830 (127.0.98.1:40439)
I20260926 01:12:31.505318 14419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17448b80435245e49447803dfc76a4cd" candidate_uuid: "5e3d9f9c88ff4617b5314f95cb3b5d47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb9887afb40444c2b400da5aeff93486"
I20260926 01:12:31.505306 14285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17448b80435245e49447803dfc76a4cd" candidate_uuid: "5e3d9f9c88ff4617b5314f95cb3b5d47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b193498b4e1a4ca091a9e021e7fa1830"
I20260926 01:12:31.505399 14285 raft_consensus.cc:3037] T 17448b80435245e49447803dfc76a4cd P b193498b4e1a4ca091a9e021e7fa1830 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:12:31.505398 14419 raft_consensus.cc:3037] T 17448b80435245e49447803dfc76a4cd P cb9887afb40444c2b400da5aeff93486 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:12:31.506312 14285 raft_consensus.cc:2454] T 17448b80435245e49447803dfc76a4cd P b193498b4e1a4ca091a9e021e7fa1830 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e3d9f9c88ff4617b5314f95cb3b5d47 in term 1.
I20260926 01:12:31.506312 14419 raft_consensus.cc:2454] T 17448b80435245e49447803dfc76a4cd P cb9887afb40444c2b400da5aeff93486 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e3d9f9c88ff4617b5314f95cb3b5d47 in term 1.
I20260926 01:12:31.506465 14488 leader_election.cc:302] T 17448b80435245e49447803dfc76a4cd P 5e3d9f9c88ff4617b5314f95cb3b5d47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e3d9f9c88ff4617b5314f95cb3b5d47, cb9887afb40444c2b400da5aeff93486; no voters:
I20260926 01:12:31.506559 14617 raft_consensus.cc:2781] T 17448b80435245e49447803dfc76a4cd P 5e3d9f9c88ff4617b5314f95cb3b5d47 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:12:31.506706 14617 raft_consensus.cc:686] T 17448b80435245e49447803dfc76a4cd P 5e3d9f9c88ff4617b5314f95cb3b5d47 [term 1 LEADER]: Becoming Leader. State: Replica: 5e3d9f9c88ff4617b5314f95cb3b5d47, State: Running, Role: LEADER
I20260926 01:12:31.506785 14617 consensus_queue.cc:237] T 17448b80435245e49447803dfc76a4cd P 5e3d9f9c88ff4617b5314f95cb3b5d47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb9887afb40444c2b400da5aeff93486" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40171 } } peers { permanent_uuid: "b193498b4e1a4ca091a9e021e7fa1830" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40439 } } peers { permanent_uuid: "5e3d9f9c88ff4617b5314f95cb3b5d47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45841 } }
I20260926 01:12:31.507625 14140 catalog_manager.cc:5495] T 17448b80435245e49447803dfc76a4cd P 5e3d9f9c88ff4617b5314f95cb3b5d47 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e3d9f9c88ff4617b5314f95cb3b5d47 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "5e3d9f9c88ff4617b5314f95cb3b5d47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb9887afb40444c2b400da5aeff93486" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b193498b4e1a4ca091a9e021e7fa1830" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e3d9f9c88ff4617b5314f95cb3b5d47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45841 } health_report { overall_health: HEALTHY } } }
W20260926 01:12:31.515805 14466 tablet.cc:2369] T 17448b80435245e49447803dfc76a4cd P cb9887afb40444c2b400da5aeff93486: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 01:12:31.519418 14612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:31.517125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38450 (local address 127.0.98.3:45841)
0926 01:12:31.517179 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 01:12:31.517181 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:12:31.517188 (+ 7us) server_negotiation.cc:407] Connection header received
0926 01:12:31.517220 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:12:31.517221 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:12:31.517241 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:12:31.517301 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:12:31.517400 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:12:31.517907 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:12:31.518632 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:12:31.518845 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:12:31.518914 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:12:31.519097 (+ 183us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:12:31.519213 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:12:31.519285 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 01:12:31.519305 (+ 20us) server_negotiation.cc:299] Negotiation successful
0926 01:12:31.519327 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 01:12:31.521399 14419 raft_consensus.cc:1261] T 17448b80435245e49447803dfc76a4cd P cb9887afb40444c2b400da5aeff93486 [term 1 FOLLOWER]: Refusing update from remote peer 5e3d9f9c88ff4617b5314f95cb3b5d47: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:12:31.521399 14285 raft_consensus.cc:1261] T 17448b80435245e49447803dfc76a4cd P b193498b4e1a4ca091a9e021e7fa1830 [term 1 FOLLOWER]: Refusing update from remote peer 5e3d9f9c88ff4617b5314f95cb3b5d47: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:12:31.521752 14623 consensus_queue.cc:1035] T 17448b80435245e49447803dfc76a4cd P 5e3d9f9c88ff4617b5314f95cb3b5d47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb9887afb40444c2b400da5aeff93486" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:12:31.521907 14617 consensus_queue.cc:1035] T 17448b80435245e49447803dfc76a4cd P 5e3d9f9c88ff4617b5314f95cb3b5d47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b193498b4e1a4ca091a9e021e7fa1830" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:12:31.524629 14628 mvcc.cc:205] Tried to move back new op lower bound from 7333417580628746240 to 7333417580572250112. Current Snapshot: MvccSnapshot[applied={T|T < 7333417580628746240}]
I20260926 01:12:31.524736 14627 mvcc.cc:205] Tried to move back new op lower bound from 7333417580628746240 to 7333417580572250112. Current Snapshot: MvccSnapshot[applied={T|T < 7333417580628746240}]
I20260926 01:12:31.526213 14139 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59326:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:12:31.526305 14139 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59326:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:12:31.527014 14139 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 396a0b42495f4fd9a266d7c98fa4012f: Sending DeleteTablet for 3 replicas of tablet 17448b80435245e49447803dfc76a4cd
I20260926 01:12:31.527323 14265 tablet_service.cc:1514] Processing DeleteTablet for tablet 17448b80435245e49447803dfc76a4cd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:12:31 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34428
I20260926 01:12:31.527328 14533 tablet_service.cc:1514] Processing DeleteTablet for tablet 17448b80435245e49447803dfc76a4cd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:12:31 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38434
I20260926 01:12:31.527539 14636 tablet_replica.cc:330] T 17448b80435245e49447803dfc76a4cd P b193498b4e1a4ca091a9e021e7fa1830: stopping tablet replica
I20260926 01:12:31.527539 14399 tablet_service.cc:1514] Processing DeleteTablet for tablet 17448b80435245e49447803dfc76a4cd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:12:31 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33088
I20260926 01:12:31.527565 14635 tablet_replica.cc:330] T 17448b80435245e49447803dfc76a4cd P 5e3d9f9c88ff4617b5314f95cb3b5d47: stopping tablet replica
I20260926 01:12:31.527626 14636 raft_consensus.cc:2229] T 17448b80435245e49447803dfc76a4cd P b193498b4e1a4ca091a9e021e7fa1830 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:12:31.527655 14635 raft_consensus.cc:2229] T 17448b80435245e49447803dfc76a4cd P 5e3d9f9c88ff4617b5314f95cb3b5d47 [term 1 LEADER]: Raft consensus shutting down.
I20260926 01:12:31.527722 14636 raft_consensus.cc:2258] T 17448b80435245e49447803dfc76a4cd P b193498b4e1a4ca091a9e021e7fa1830 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:12:31.527751 14635 raft_consensus.cc:2258] T 17448b80435245e49447803dfc76a4cd P 5e3d9f9c88ff4617b5314f95cb3b5d47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:12:31.527748 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 14202
I20260926 01:12:31.528719 14635 ts_tablet_manager.cc:1905] T 17448b80435245e49447803dfc76a4cd P 5e3d9f9c88ff4617b5314f95cb3b5d47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:12:31.530736 14635 ts_tablet_manager.cc:1918] T 17448b80435245e49447803dfc76a4cd P 5e3d9f9c88ff4617b5314f95cb3b5d47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:12:31.530817 14635 log.cc:1196] T 17448b80435245e49447803dfc76a4cd P 5e3d9f9c88ff4617b5314f95cb3b5d47: Deleting WAL directory at /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-2/wal/wals/17448b80435245e49447803dfc76a4cd
I20260926 01:12:31.531080 14635 ts_tablet_manager.cc:1939] T 17448b80435245e49447803dfc76a4cd P 5e3d9f9c88ff4617b5314f95cb3b5d47: Deleting consensus metadata
I20260926 01:12:31.531629 14126 catalog_manager.cc:4841] TS 5e3d9f9c88ff4617b5314f95cb3b5d47 (127.0.98.3:45841): tablet 17448b80435245e49447803dfc76a4cd (table test-table [id=3e310cb802d84d10a4e406f91ccbfec8]) successfully deleted
W20260926 01:12:31.533159 14125 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:40439 (error 108)
I20260926 01:12:31.533666 14607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:31.533532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40439 (local address 127.0.0.1:34436)
0926 01:12:31.533588 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0926 01:12:31.533628 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260926 01:12:31.533664 14637 tablet_replica.cc:330] T 17448b80435245e49447803dfc76a4cd P cb9887afb40444c2b400da5aeff93486: stopping tablet replica
I20260926 01:12:31.533664 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 14336
W20260926 01:12:31.534405 14125 catalog_manager.cc:4584] TS b193498b4e1a4ca091a9e021e7fa1830 (127.0.98.1:40439): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 17448b80435245e49447803dfc76a4cd: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40439: connect: Connection refused (error 111)
I20260926 01:12:31.533790 14637 raft_consensus.cc:2229] T 17448b80435245e49447803dfc76a4cd P cb9887afb40444c2b400da5aeff93486 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:12:31.538452 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 14470
I20260926 01:12:31.538633 14607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:31.538487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40171 (local address 127.0.0.1:33102)
0926 01:12:31.538562 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0926 01:12:31.538596 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:40171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260926 01:12:31.542724 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 14108
I20260926 01:12:31.547627 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32949
--webserver_interface=127.0.98.62
--webserver_port=46799
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:32949
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:12:31.634181 14639 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:12:31.634407 14639 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:12:31.634430 14639 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:12:31.635677 14639 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:12:31.635727 14639 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:12:31.635741 14639 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:12:31.635752 14639 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:12:31.635763 14639 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:12:31.637356 14639 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:32949
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32949
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=46799
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14639
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:12:31.637615 14639 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:12:31.637895 14639 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:12:31.647500 14645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:12:31.647511 14646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:12:31.647593 14639 server_base.cc:1034] running on GCE node
W20260926 01:12:31.647519 14648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:12:31.648053 14639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:12:31.648376 14639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:12:31.649559 14639 hybrid_clock.cc:648] HybridClock initialized: now 1790385151649526 us; error 47 us; skew 500 ppm
Sep 26 01:12:31 dist-test-slave-7b22 krb5kdc[14095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385151, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:12:31.653304 14639 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 01:12:31.653791 14639 webserver.cc:460] Webserver started at http://127.0.98.62:46799/ using document root <none> and password file <none>
I20260926 01:12:31.654112 14639 fs_manager.cc:359] Metadata directory not provided
I20260926 01:12:31.654191 14639 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:12:31.655671 14639 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 01:12:31.656497 14655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:12:31.656700 14639 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 01:12:31.656795 14639 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/master-0/wal
uuid: "396a0b42495f4fd9a266d7c98fa4012f"
format_stamp: "Formatted at 2026-09-26 01:12:30 on dist-test-slave-7b22"
server_key: "34eef3d71352829a61341ec88d1333f2"
server_key_iv: "a3904234ff1334e45601d2cf3db600ba"
server_key_version: "encryptionkey@0"
I20260926 01:12:31.657292 14639 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:12:31.683818 14639 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:12:31.686278 14639 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:12:31.686403 14639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:12:31.690559 14639 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:32949
I20260926 01:12:31.690624 14707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:32949 every 8 connection(s)
I20260926 01:12:31.690990 14639 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
I20260926 01:12:31.692623 14708 sys_catalog.cc:263] Verifying existing consensus state
I20260926 01:12:31.693073 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 14639
I20260926 01:12:31.693323 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:40439
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:32949
--webserver_port=43983
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 01:12:31.693327 14708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 396a0b42495f4fd9a266d7c98fa4012f: Bootstrap starting.
I20260926 01:12:31.702006 14708 log.cc:824] T 00000000000000000000000000000000 P 396a0b42495f4fd9a266d7c98fa4012f: Log is configured to *not* fsync() on all Append() calls
I20260926 01:12:31.704061 14708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 396a0b42495f4fd9a266d7c98fa4012f: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 01:12:31.704329 14708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 396a0b42495f4fd9a266d7c98fa4012f: Bootstrap complete.
I20260926 01:12:31.706734 14708 raft_consensus.cc:348] T 00000000000000000000000000000000 P 396a0b42495f4fd9a266d7c98fa4012f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "396a0b42495f4fd9a266d7c98fa4012f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32949 } }
I20260926 01:12:31.707002 14708 raft_consensus.cc:729] T 00000000000000000000000000000000 P 396a0b42495f4fd9a266d7c98fa4012f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 396a0b42495f4fd9a266d7c98fa4012f, State: Initialized, Role: FOLLOWER
I20260926 01:12:31.707165 14708 consensus_queue.cc:260] T 00000000000000000000000000000000 P 396a0b42495f4fd9a266d7c98fa4012f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "396a0b42495f4fd9a266d7c98fa4012f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32949 } }
I20260926 01:12:31.707242 14708 raft_consensus.cc:388] T 00000000000000000000000000000000 P 396a0b42495f4fd9a266d7c98fa4012f [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:12:31.707297 14708 raft_consensus.cc:482] T 00000000000000000000000000000000 P 396a0b42495f4fd9a266d7c98fa4012f [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:12:31.707347 14708 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 396a0b42495f4fd9a266d7c98fa4012f [term 1 FOLLOWER]: Advancing to term 2
I20260926 01:12:31.708442 14708 raft_consensus.cc:504] T 00000000000000000000000000000000 P 396a0b42495f4fd9a266d7c98fa4012f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "396a0b42495f4fd9a266d7c98fa4012f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32949 } }
I20260926 01:12:31.708546 14708 leader_election.cc:302] T 00000000000000000000000000000000 P 396a0b42495f4fd9a266d7c98fa4012f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 396a0b42495f4fd9a266d7c98fa4012f; no voters:
I20260926 01:12:31.708715 14708 leader_election.cc:288] T 00000000000000000000000000000000 P 396a0b42495f4fd9a266d7c98fa4012f [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 01:12:31.708766 14712 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 396a0b42495f4fd9a266d7c98fa4012f [term 2 FOLLOWER]: Leader election won for term 2
I20260926 01:12:31.708912 14712 raft_consensus.cc:686] T 00000000000000000000000000000000 P 396a0b42495f4fd9a266d7c98fa4012f [term 2 LEADER]: Becoming Leader. State: Replica: 396a0b42495f4fd9a266d7c98fa4012f, State: Running, Role: LEADER
I20260926 01:12:31.709007 14708 sys_catalog.cc:564] T 00000000000000000000000000000000 P 396a0b42495f4fd9a266d7c98fa4012f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:12:31.709054 14712 consensus_queue.cc:237] T 00000000000000000000000000000000 P 396a0b42495f4fd9a266d7c98fa4012f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "396a0b42495f4fd9a266d7c98fa4012f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32949 } }
I20260926 01:12:31.709362 14713 sys_catalog.cc:455] T 00000000000000000000000000000000 P 396a0b42495f4fd9a266d7c98fa4012f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "396a0b42495f4fd9a266d7c98fa4012f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "396a0b42495f4fd9a266d7c98fa4012f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32949 } } }
I20260926 01:12:31.709465 14713 sys_catalog.cc:458] T 00000000000000000000000000000000 P 396a0b42495f4fd9a266d7c98fa4012f [sys.catalog]: This master's current role is: LEADER
I20260926 01:12:31.709652 14712 sys_catalog.cc:455] T 00000000000000000000000000000000 P 396a0b42495f4fd9a266d7c98fa4012f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 396a0b42495f4fd9a266d7c98fa4012f. Latest consensus state: current_term: 2 leader_uuid: "396a0b42495f4fd9a266d7c98fa4012f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "396a0b42495f4fd9a266d7c98fa4012f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32949 } } }
I20260926 01:12:31.709703 14712 sys_catalog.cc:458] T 00000000000000000000000000000000 P 396a0b42495f4fd9a266d7c98fa4012f [sys.catalog]: This master's current role is: LEADER
I20260926 01:12:31.709693 14717 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:12:31.710711 14717 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:12:31.710851 14717 catalog_manager.cc:1259] Loaded cluster ID: 9c532ce79c0e4442ad110b86d288c165
I20260926 01:12:31.711032 14717 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 01:12:31.711294 14717 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260926 01:12:31.715901 14717 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260926 01:12:31.718786 14717 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:12:31.719051 14717 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 396a0b42495f4fd9a266d7c98fa4012f: Loaded TSK: 0
I20260926 01:12:31.719281 14717 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 01:12:31.781742 14710 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:12:31.782007 14710 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:12:31.782043 14710 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:12:31.783298 14710 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:12:31.783347 14710 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:12:31.783385 14710 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 01:12:31.784962 14710 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:40439
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=43983
--tserver_master_addrs=127.0.98.62:32949
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14710
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:12:31.785233 14710 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:12:31.785460 14710 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:12:31.790568 14735 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:12:31.790661 14736 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:12:31.790551 14738 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:12:31.791033 14710 server_base.cc:1034] running on GCE node
I20260926 01:12:31.791240 14710 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:12:31.791486 14710 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:12:31.792713 14710 hybrid_clock.cc:648] HybridClock initialized: now 1790385151792686 us; error 42 us; skew 500 ppm
Sep 26 01:12:31 dist-test-slave-7b22 krb5kdc[14095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385151, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:12:31.795483 14710 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 01:12:31.795989 14710 webserver.cc:460] Webserver started at http://127.0.98.1:43983/ using document root <none> and password file <none>
I20260926 01:12:31.796216 14710 fs_manager.cc:359] Metadata directory not provided
I20260926 01:12:31.796274 14710 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:12:31.797657 14710 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:12:31.798378 14745 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:12:31.798605 14710 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 01:12:31.798714 14710 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-0/wal
uuid: "b193498b4e1a4ca091a9e021e7fa1830"
format_stamp: "Formatted at 2026-09-26 01:12:31 on dist-test-slave-7b22"
server_key: "fdd3b728c1189ff7ea60b8bc9bd976b7"
server_key_iv: "c2490517d7a60083a59be57bc4e4ed69"
server_key_version: "encryptionkey@0"
I20260926 01:12:31.799175 14710 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:12:31.813473 14710 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:12:31.816259 14710 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:12:31.816428 14710 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:12:31.816685 14710 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:12:31.817231 14752 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 01:12:31.818977 14710 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 01:12:31.819036 14710 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260926 01:12:31.819062 14710 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 01:12:31.819722 14710 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 01:12:31.819766 14710 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260926 01:12:31.819834 14752 tablet_bootstrap.cc:492] T 17448b80435245e49447803dfc76a4cd P b193498b4e1a4ca091a9e021e7fa1830: Bootstrap starting.
I20260926 01:12:31.827072 14710 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40439
I20260926 01:12:31.827730 14710 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-0/data/info.pb
I20260926 01:12:31.827742 14859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40439 every 8 connection(s)
I20260926 01:12:31.829051 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 14710
I20260926 01:12:31.829563 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:40171
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:32949
--webserver_port=37981
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 01:12:31.831415 14752 log.cc:824] T 17448b80435245e49447803dfc76a4cd P b193498b4e1a4ca091a9e021e7fa1830: Log is configured to *not* fsync() on all Append() calls
I20260926 01:12:31.834583 14752 tablet_bootstrap.cc:492] T 17448b80435245e49447803dfc76a4cd P b193498b4e1a4ca091a9e021e7fa1830: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 01:12:31.835027 14752 tablet_bootstrap.cc:492] T 17448b80435245e49447803dfc76a4cd P b193498b4e1a4ca091a9e021e7fa1830: Bootstrap complete.
I20260926 01:12:31.836155 14752 ts_tablet_manager.cc:1397] T 17448b80435245e49447803dfc76a4cd P b193498b4e1a4ca091a9e021e7fa1830: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.007s
I20260926 01:12:31.837616 14752 raft_consensus.cc:348] T 17448b80435245e49447803dfc76a4cd P b193498b4e1a4ca091a9e021e7fa1830 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb9887afb40444c2b400da5aeff93486" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40171 } } peers { permanent_uuid: "b193498b4e1a4ca091a9e021e7fa1830" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40439 } } peers { permanent_uuid: "5e3d9f9c88ff4617b5314f95cb3b5d47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45841 } }
I20260926 01:12:31.838030 14752 raft_consensus.cc:729] T 17448b80435245e49447803dfc76a4cd P b193498b4e1a4ca091a9e021e7fa1830 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b193498b4e1a4ca091a9e021e7fa1830, State: Initialized, Role: FOLLOWER
I20260926 01:12:31.838199 14752 consensus_queue.cc:260] T 17448b80435245e49447803dfc76a4cd P b193498b4e1a4ca091a9e021e7fa1830 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb9887afb40444c2b400da5aeff93486" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40171 } } peers { permanent_uuid: "b193498b4e1a4ca091a9e021e7fa1830" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40439 } } peers { permanent_uuid: "5e3d9f9c88ff4617b5314f95cb3b5d47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45841 } }
I20260926 01:12:31.838496 14752 ts_tablet_manager.cc:1428] T 17448b80435245e49447803dfc76a4cd P b193498b4e1a4ca091a9e021e7fa1830: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
Sep 26 01:12:31 dist-test-slave-7b22 krb5kdc[14095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385151, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:12:31.847307 14864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:31.833262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32949 (local address 127.0.98.1:33911)
0926 01:12:31.833962 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0926 01:12:31.834012 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 01:12:31.834626 (+ 614us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:12:31.835405 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:12:31.835409 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:12:31.835774 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:12:31.836089 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:12:31.836097 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:12:31.837862 (+ 1765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:12:31.837864 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:12:31.839178 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:12:31.839182 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:12:31.840057 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:12:31.840059 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:12:31.840211 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:12:31.840854 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:12:31.840865 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:12:31.842786 (+ 1921us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:12:31.845065 (+ 2279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:12:31.845076 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:12:31.845078 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:12:31.845403 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:12:31.845727 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:12:31.845728 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:12:31.845729 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:12:31.845923 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:12:31.846438 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:12:31.846444 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:12:31.846713 (+ 269us) client_negotiation.cc:769] Sending connection context
0926 01:12:31.846833 (+ 120us) client_negotiation.cc:240] Negotiation successful
0926 01:12:31.847074 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":644,"thread_start_us":86,"threads_started":1}
I20260926 01:12:31.847362 14865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:31.833587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33911 (local address 127.0.98.62:32949)
0926 01:12:31.834849 (+ 1262us) server_negotiation.cc:206] Beginning negotiation
0926 01:12:31.834859 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 01:12:31.834915 (+ 56us) server_negotiation.cc:407] Connection header received
0926 01:12:31.835059 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:12:31.835066 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:12:31.835124 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:12:31.835260 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:12:31.836236 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:12:31.837713 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:12:31.839291 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:12:31.839933 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:12:31.840068 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:12:31.842953 (+ 2885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:12:31.842958 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:12:31.842958 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:12:31.842974 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:12:31.844961 (+ 1987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:12:31.845504 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:12:31.845507 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:12:31.845508 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:12:31.845630 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:12:31.846039 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:12:31.846040 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:12:31.846040 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:12:31.846367 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:12:31.846454 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 01:12:31.847002 (+ 548us) server_negotiation.cc:299] Negotiation successful
0926 01:12:31.847168 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1205,"thread_start_us":87,"threads_started":1}
I20260926 01:12:31.848405 14860 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32949
I20260926 01:12:31.848508 14860 heartbeater.cc:464] Registering TS with master...
I20260926 01:12:31.848692 14860 heartbeater.cc:511] Master 127.0.98.62:32949 requested a full tablet report, sending...
I20260926 01:12:31.849607 14672 ts_manager.cc:194] Registered new tserver with Master: b193498b4e1a4ca091a9e021e7fa1830 (127.0.98.1:40439)
I20260926 01:12:31.851166 14672 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:33911
I20260926 01:12:31.852218 14860 heartbeater.cc:503] Master 127.0.98.62:32949 was elected leader, sending a full tablet report...
Sep 26 01:12:31 dist-test-slave-7b22 krb5kdc[14095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385151, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 01:12:31.860548 14867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:31.850575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40439 (local address 127.0.0.1:34440)
0926 01:12:31.851503 (+ 928us) negotiation.cc:107] Waiting for socket to connect
0926 01:12:31.851548 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 01:12:31.851627 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:12:31.851897 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:12:31.851901 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:12:31.852240 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:12:31.852303 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:12:31.852306 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:12:31.853787 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:12:31.853789 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:12:31.854781 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:12:31.854784 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:12:31.855378 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:12:31.855382 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:12:31.855447 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:12:31.855936 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:12:31.855948 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:12:31.857395 (+ 1447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:12:31.859168 (+ 1773us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:12:31.859175 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:12:31.859178 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:12:31.859439 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:12:31.859697 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:12:31.859698 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:12:31.859698 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:12:31.859766 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:12:31.860141 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:12:31.860148 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:12:31.860210 (+ 62us) client_negotiation.cc:769] Sending connection context
0926 01:12:31.860296 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 01:12:31.860433 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":901,"thread_start_us":133,"threads_started":1}
I20260926 01:12:31.860679 14868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:31.850882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34440 (local address 127.0.98.1:40439)
0926 01:12:31.851076 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 01:12:31.851081 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 01:12:31.851708 (+ 627us) server_negotiation.cc:407] Connection header received
0926 01:12:31.851754 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:12:31.851756 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:12:31.851773 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:12:31.851830 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:12:31.852390 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:12:31.853693 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:12:31.854861 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:12:31.855296 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:12:31.855371 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:12:31.857527 (+ 2156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:12:31.857535 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:12:31.857535 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:12:31.857542 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:12:31.859079 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:12:31.859558 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:12:31.859561 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:12:31.859561 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:12:31.859630 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:12:31.859919 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:12:31.859920 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:12:31.859920 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:12:31.860079 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:12:31.860142 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 01:12:31.860419 (+ 277us) server_negotiation.cc:299] Negotiation successful
0926 01:12:31.860582 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":78,"threads_started":1}
I20260926 01:12:31.861243 14794 tablet_service.cc:1514] Processing DeleteTablet for tablet 17448b80435245e49447803dfc76a4cd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:12:31 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34440
I20260926 01:12:31.861840 14869 tablet_replica.cc:330] stopping tablet replica
I20260926 01:12:31.861958 14869 raft_consensus.cc:2229] T 17448b80435245e49447803dfc76a4cd P b193498b4e1a4ca091a9e021e7fa1830 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:12:31.862028 14869 raft_consensus.cc:2258] T 17448b80435245e49447803dfc76a4cd P b193498b4e1a4ca091a9e021e7fa1830 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:12:31.862349 14869 ts_tablet_manager.cc:1905] T 17448b80435245e49447803dfc76a4cd P b193498b4e1a4ca091a9e021e7fa1830: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:12:31.864804 14869 ts_tablet_manager.cc:1918] T 17448b80435245e49447803dfc76a4cd P b193498b4e1a4ca091a9e021e7fa1830: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:12:31.864882 14869 log.cc:1196] T 17448b80435245e49447803dfc76a4cd P b193498b4e1a4ca091a9e021e7fa1830: Deleting WAL directory at /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-0/wal/wals/17448b80435245e49447803dfc76a4cd
I20260926 01:12:31.865085 14869 ts_tablet_manager.cc:1939] T 17448b80435245e49447803dfc76a4cd P b193498b4e1a4ca091a9e021e7fa1830: Deleting consensus metadata
I20260926 01:12:31.865665 14657 catalog_manager.cc:4841] TS b193498b4e1a4ca091a9e021e7fa1830 (127.0.98.1:40439): tablet 17448b80435245e49447803dfc76a4cd (table test-table [id=3e310cb802d84d10a4e406f91ccbfec8]) successfully deleted
W20260926 01:12:31.920236 14862 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:12:31.920454 14862 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:12:31.920478 14862 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:12:31.921654 14862 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:12:31.921700 14862 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:12:31.921727 14862 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 01:12:31.923203 14862 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:40171
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=37981
--tserver_master_addrs=127.0.98.62:32949
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14862
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:12:31.923422 14862 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:12:31.923645 14862 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:12:31.928530 14875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:12:31.928526 14877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:12:31.928580 14874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:12:31.928593 14862 server_base.cc:1034] running on GCE node
I20260926 01:12:31.928983 14862 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:12:31.929214 14862 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:12:31.930377 14862 hybrid_clock.cc:648] HybridClock initialized: now 1790385151930353 us; error 36 us; skew 500 ppm
Sep 26 01:12:31 dist-test-slave-7b22 krb5kdc[14095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385151, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:12:31.932967 14862 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 01:12:31.933405 14862 webserver.cc:460] Webserver started at http://127.0.98.2:37981/ using document root <none> and password file <none>
I20260926 01:12:31.933598 14862 fs_manager.cc:359] Metadata directory not provided
I20260926 01:12:31.933660 14862 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:12:31.934993 14862 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:12:31.935570 14884 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:12:31.935719 14862 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 01:12:31.935799 14862 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-1/wal
uuid: "cb9887afb40444c2b400da5aeff93486"
format_stamp: "Formatted at 2026-09-26 01:12:31 on dist-test-slave-7b22"
server_key: "9349ff8e45cff9d62e0eb3b58ca443e3"
server_key_iv: "73f73772a4aeec2ab5af356f748f85df"
server_key_version: "encryptionkey@0"
I20260926 01:12:31.936164 14862 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:12:31.966444 14862 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:12:31.968622 14862 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:12:31.968780 14862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:12:31.969025 14862 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:12:31.969424 14891 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 01:12:31.970980 14862 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 01:12:31.971038 14862 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260926 01:12:31.971073 14862 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 01:12:31.971719 14862 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 01:12:31.971753 14862 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260926 01:12:31.971812 14891 tablet_bootstrap.cc:492] T 17448b80435245e49447803dfc76a4cd P cb9887afb40444c2b400da5aeff93486: Bootstrap starting.
I20260926 01:12:31.977474 14862 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40171
I20260926 01:12:31.977617 14998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40171 every 8 connection(s)
I20260926 01:12:31.977938 14862 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-1/data/info.pb
I20260926 01:12:31.981580 14891 log.cc:824] T 17448b80435245e49447803dfc76a4cd P cb9887afb40444c2b400da5aeff93486: Log is configured to *not* fsync() on all Append() calls
I20260926 01:12:31.984207 14891 tablet_bootstrap.cc:492] T 17448b80435245e49447803dfc76a4cd P cb9887afb40444c2b400da5aeff93486: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 01:12:31.984516 14891 tablet_bootstrap.cc:492] T 17448b80435245e49447803dfc76a4cd P cb9887afb40444c2b400da5aeff93486: Bootstrap complete.
Sep 26 01:12:31 dist-test-slave-7b22 krb5kdc[14095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385151, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:12:31.985311 14891 ts_tablet_manager.cc:1397] T 17448b80435245e49447803dfc76a4cd P cb9887afb40444c2b400da5aeff93486: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.004s
I20260926 01:12:31.986383 14891 raft_consensus.cc:348] T 17448b80435245e49447803dfc76a4cd P cb9887afb40444c2b400da5aeff93486 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb9887afb40444c2b400da5aeff93486" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40171 } } peers { permanent_uuid: "b193498b4e1a4ca091a9e021e7fa1830" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40439 } } peers { permanent_uuid: "5e3d9f9c88ff4617b5314f95cb3b5d47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45841 } }
I20260926 01:12:31.986660 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 14862
I20260926 01:12:31.986675 14891 raft_consensus.cc:729] T 17448b80435245e49447803dfc76a4cd P cb9887afb40444c2b400da5aeff93486 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb9887afb40444c2b400da5aeff93486, State: Initialized, Role: FOLLOWER
I20260926 01:12:31.986865 14891 consensus_queue.cc:260] T 17448b80435245e49447803dfc76a4cd P cb9887afb40444c2b400da5aeff93486 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb9887afb40444c2b400da5aeff93486" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40171 } } peers { permanent_uuid: "b193498b4e1a4ca091a9e021e7fa1830" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40439 } } peers { permanent_uuid: "5e3d9f9c88ff4617b5314f95cb3b5d47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45841 } }
I20260926 01:12:31.986972 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:45841
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:32949
--webserver_port=44729
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 01:12:31.987178 14891 ts_tablet_manager.cc:1428] T 17448b80435245e49447803dfc76a4cd P cb9887afb40444c2b400da5aeff93486: Time spent starting tablet: real 0.002s user 0.002s sys 0.003s
I20260926 01:12:31.988700 14865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:31.978711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33979 (local address 127.0.98.62:32949)
0926 01:12:31.978777 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 01:12:31.978780 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:12:31.978976 (+ 196us) server_negotiation.cc:407] Connection header received
0926 01:12:31.979712 (+ 736us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:12:31.979714 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:12:31.979729 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:12:31.979787 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:12:31.980766 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:12:31.981532 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:12:31.982927 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:12:31.983137 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:12:31.983204 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:12:31.985633 (+ 2429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:12:31.985634 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:12:31.985635 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:12:31.985643 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:12:31.987230 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:12:31.987705 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:12:31.987706 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:12:31.987707 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:12:31.987760 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:12:31.988023 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:12:31.988024 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:12:31.988024 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:12:31.988152 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:12:31.988215 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 01:12:31.988513 (+ 298us) server_negotiation.cc:299] Negotiation successful
0926 01:12:31.988608 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 01:12:31.988763 15001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:31.978648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32949 (local address 127.0.98.2:33979)
0926 01:12:31.978831 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0926 01:12:31.978883 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 01:12:31.979580 (+ 697us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:12:31.979881 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:12:31.979886 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:12:31.980329 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:12:31.980636 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:12:31.980645 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:12:31.981664 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:12:31.981665 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:12:31.982795 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:12:31.982798 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:12:31.983245 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:12:31.983247 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:12:31.983400 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:12:31.983936 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:12:31.983945 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:12:31.985528 (+ 1583us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:12:31.987330 (+ 1802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:12:31.987336 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:12:31.987338 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:12:31.987630 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:12:31.987828 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:12:31.987829 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:12:31.987829 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:12:31.987969 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:12:31.988214 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:12:31.988219 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:12:31.988397 (+ 178us) client_negotiation.cc:769] Sending connection context
0926 01:12:31.988482 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 01:12:31.988614 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":84,"threads_started":1}
I20260926 01:12:31.989212 14999 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32949
I20260926 01:12:31.989301 14999 heartbeater.cc:464] Registering TS with master...
I20260926 01:12:31.989465 14999 heartbeater.cc:511] Master 127.0.98.62:32949 requested a full tablet report, sending...
I20260926 01:12:31.990022 14672 ts_manager.cc:194] Registered new tserver with Master: cb9887afb40444c2b400da5aeff93486 (127.0.98.2:40171)
I20260926 01:12:31.990592 14672 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:33979
I20260926 01:12:31.991866 14999 heartbeater.cc:503] Master 127.0.98.62:32949 was elected leader, sending a full tablet report...
Sep 26 01:12:31 dist-test-slave-7b22 krb5kdc[14095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385151, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260926 01:12:31.997164 14867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:31.990304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40171 (local address 127.0.0.1:33116)
0926 01:12:31.990352 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0926 01:12:31.990362 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 01:12:31.990396 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:12:31.990680 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:12:31.990682 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:12:31.990843 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:12:31.990898 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:12:31.990900 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:12:31.991861 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:12:31.991862 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:12:31.992426 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:12:31.992429 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:12:31.992927 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:12:31.992930 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:12:31.993000 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:12:31.993398 (+ 398us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:12:31.993404 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:12:31.994550 (+ 1146us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:12:31.996123 (+ 1573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:12:31.996124 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:12:31.996125 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:12:31.996324 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:12:31.996509 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:12:31.996510 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:12:31.996511 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:12:31.996571 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:12:31.996831 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:12:31.996832 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:12:31.996882 (+ 50us) client_negotiation.cc:769] Sending connection context
0926 01:12:31.996945 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 01:12:31.997056 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260926 01:12:31.997164 15005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:31.990384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33116 (local address 127.0.98.2:40171)
0926 01:12:31.990532 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0926 01:12:31.990537 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 01:12:31.990546 (+ 9us) server_negotiation.cc:407] Connection header received
0926 01:12:31.990578 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:12:31.990579 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:12:31.990593 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:12:31.990659 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:12:31.991037 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:12:31.991776 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:12:31.992500 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:12:31.992811 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:12:31.992924 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:12:31.994627 (+ 1703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:12:31.994636 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:12:31.994636 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:12:31.994644 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:12:31.996039 (+ 1395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:12:31.996397 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:12:31.996400 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:12:31.996400 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:12:31.996453 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:12:31.996618 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:12:31.996619 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:12:31.996620 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:12:31.996771 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:12:31.996850 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 01:12:31.996984 (+ 134us) server_negotiation.cc:299] Negotiation successful
0926 01:12:31.997073 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":82,"threads_started":1}
I20260926 01:12:31.997635 14933 tablet_service.cc:1514] Processing DeleteTablet for tablet 17448b80435245e49447803dfc76a4cd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:12:31 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33116
I20260926 01:12:31.998306 15006 tablet_replica.cc:330] stopping tablet replica
I20260926 01:12:31.998425 15006 raft_consensus.cc:2229] T 17448b80435245e49447803dfc76a4cd P cb9887afb40444c2b400da5aeff93486 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:12:31.998499 15006 raft_consensus.cc:2258] T 17448b80435245e49447803dfc76a4cd P cb9887afb40444c2b400da5aeff93486 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:12:31.998803 15006 ts_tablet_manager.cc:1905] T 17448b80435245e49447803dfc76a4cd P cb9887afb40444c2b400da5aeff93486: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:12:32.001098 15006 ts_tablet_manager.cc:1918] T 17448b80435245e49447803dfc76a4cd P cb9887afb40444c2b400da5aeff93486: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:12:32.001186 15006 log.cc:1196] T 17448b80435245e49447803dfc76a4cd P cb9887afb40444c2b400da5aeff93486: Deleting WAL directory at /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-1/wal/wals/17448b80435245e49447803dfc76a4cd
I20260926 01:12:32.001389 15006 ts_tablet_manager.cc:1939] T 17448b80435245e49447803dfc76a4cd P cb9887afb40444c2b400da5aeff93486: Deleting consensus metadata
I20260926 01:12:32.001952 14658 catalog_manager.cc:4841] TS cb9887afb40444c2b400da5aeff93486 (127.0.98.2:40171): tablet 17448b80435245e49447803dfc76a4cd (table test-table [id=3e310cb802d84d10a4e406f91ccbfec8]) successfully deleted
W20260926 01:12:32.070317 15004 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:12:32.070560 15004 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:12:32.070591 15004 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:12:32.071811 15004 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:12:32.071848 15004 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:12:32.071874 15004 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 01:12:32.073381 15004 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:45841
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=44729
--tserver_master_addrs=127.0.98.62:32949
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15004
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:12:32.073593 15004 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:12:32.073784 15004 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:12:32.078750 15012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:12:32.078750 15011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:12:32.078926 15004 server_base.cc:1034] running on GCE node
W20260926 01:12:32.078744 15014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:12:32.079236 15004 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:12:32.079453 15004 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:12:32.080612 15004 hybrid_clock.cc:648] HybridClock initialized: now 1790385152080598 us; error 30 us; skew 500 ppm
Sep 26 01:12:32 dist-test-slave-7b22 krb5kdc[14095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385152, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:12:32.083128 15004 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 01:12:32.083539 15004 webserver.cc:460] Webserver started at http://127.0.98.3:44729/ using document root <none> and password file <none>
I20260926 01:12:32.083729 15004 fs_manager.cc:359] Metadata directory not provided
I20260926 01:12:32.083781 15004 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:12:32.085151 15004 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:12:32.085798 15021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:12:32.085999 15004 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:12:32.086081 15004 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-2/wal
uuid: "5e3d9f9c88ff4617b5314f95cb3b5d47"
format_stamp: "Formatted at 2026-09-26 01:12:31 on dist-test-slave-7b22"
server_key: "8bfee5695dbe2c9914163b20a004de48"
server_key_iv: "1c2a4134373cdf7cf9808899feaa641f"
server_key_version: "encryptionkey@0"
I20260926 01:12:32.086438 15004 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:12:32.102860 15004 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:12:32.105182 15004 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:12:32.105301 15004 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:12:32.105531 15004 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:12:32.105813 15004 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:12:32.105885 15004 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:12:32.105914 15004 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:12:32.105937 15004 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:12:32.111541 15004 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45841
I20260926 01:12:32.111606 15134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45841 every 8 connection(s)
I20260926 01:12:32.111912 15004 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-2/data/info.pb
I20260926 01:12:32.112370 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 15004
Sep 26 01:12:32 dist-test-slave-7b22 krb5kdc[14095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385152, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:12:32.119319 15139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:32.112681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59334 (local address 127.0.98.62:32949)
0926 01:12:32.112774 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0926 01:12:32.112776 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:12:32.112913 (+ 137us) server_negotiation.cc:407] Connection header received
0926 01:12:32.112978 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:12:32.112980 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:12:32.112992 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:12:32.113057 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:12:32.113721 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:12:32.114306 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:12:32.115138 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:12:32.115385 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:12:32.115434 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:12:32.116792 (+ 1358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:12:32.116793 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:12:32.116793 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:12:32.116801 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:12:32.118280 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:12:32.118608 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:12:32.118609 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:12:32.118610 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:12:32.118668 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:12:32.118838 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:12:32.118838 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:12:32.118839 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:12:32.118953 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:12:32.119121 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0926 01:12:32.119162 (+ 41us) server_negotiation.cc:299] Negotiation successful
0926 01:12:32.119233 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74,"thread_start_us":47,"threads_started":1}
I20260926 01:12:32.121409 14865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:32.112575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45545 (local address 127.0.98.62:32949)
0926 01:12:32.112643 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 01:12:32.112645 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:12:32.112908 (+ 263us) server_negotiation.cc:407] Connection header received
0926 01:12:32.113554 (+ 646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:12:32.113555 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:12:32.113562 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:12:32.113592 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:12:32.114693 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:12:32.115201 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:12:32.116643 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:12:32.116859 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:12:32.116927 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:12:32.118962 (+ 2035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:12:32.118963 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:12:32.118963 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:12:32.118969 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:12:32.120365 (+ 1396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:12:32.120714 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:12:32.120714 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:12:32.120715 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:12:32.120756 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:12:32.120944 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:12:32.120944 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:12:32.120945 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:12:32.121035 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:12:32.121079 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0926 01:12:32.121291 (+ 212us) server_negotiation.cc:299] Negotiation successful
0926 01:12:32.121346 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 01:12:32.121636 15137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:32.112601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32949 (local address 127.0.98.3:45545)
0926 01:12:32.112775 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0926 01:12:32.112827 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 01:12:32.113454 (+ 627us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:12:32.113678 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:12:32.113682 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:12:32.114297 (+ 615us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:12:32.114538 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:12:32.114544 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:12:32.115321 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:12:32.115322 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:12:32.116545 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:12:32.116549 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:12:32.116942 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:12:32.116943 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:12:32.117054 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:12:32.117546 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:12:32.117554 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:12:32.118885 (+ 1331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:12:32.120436 (+ 1551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:12:32.120442 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:12:32.120444 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:12:32.120658 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:12:32.120797 (+ 139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:12:32.120798 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:12:32.120798 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:12:32.120887 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:12:32.121077 (+ 190us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:12:32.121080 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:12:32.121195 (+ 115us) client_negotiation.cc:769] Sending connection context
0926 01:12:32.121273 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 01:12:32.121392 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":96,"threads_started":1}
I20260926 01:12:32.122066 15135 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32949
I20260926 01:12:32.122161 15135 heartbeater.cc:464] Registering TS with master...
I20260926 01:12:32.122334 15135 heartbeater.cc:511] Master 127.0.98.62:32949 requested a full tablet report, sending...
I20260926 01:12:32.122663 14672 ts_manager.cc:194] Registered new tserver with Master: 5e3d9f9c88ff4617b5314f95cb3b5d47 (127.0.98.3:45841)
I20260926 01:12:32.123023 14672 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:45545
I20260926 01:12:32.131111 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 01:12:32.133885 14865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:32.131459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59340 (local address 127.0.98.62:32949)
0926 01:12:32.131511 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 01:12:32.131514 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:12:32.131612 (+ 98us) server_negotiation.cc:407] Connection header received
0926 01:12:32.131675 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:12:32.131676 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:12:32.131711 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:12:32.131770 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:12:32.131871 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:12:32.132371 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:12:32.133163 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:12:32.133347 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:12:32.133399 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:12:32.133578 (+ 179us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:12:32.133697 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:12:32.133755 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 01:12:32.133766 (+ 11us) server_negotiation.cc:299] Negotiation successful
0926 01:12:32.133783 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260926 01:12:32.134183 14672 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:59340:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:12:32.135826 14672 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:12:32.137965 14794 tablet_service.cc:1467] Processing CreateTablet for tablet db9d461f21454562ac5f5d627b91a45e (DEFAULT_TABLE table=test-table [id=4d39f95b7200452b8ce81c66d560a687]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:12:32.138137 14933 tablet_service.cc:1467] Processing CreateTablet for tablet db9d461f21454562ac5f5d627b91a45e (DEFAULT_TABLE table=test-table [id=4d39f95b7200452b8ce81c66d560a687]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:12:32.138183 14794 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db9d461f21454562ac5f5d627b91a45e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:12:32.138290 14933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db9d461f21454562ac5f5d627b91a45e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:12:32.139845 14752 tablet_bootstrap.cc:492] T db9d461f21454562ac5f5d627b91a45e P b193498b4e1a4ca091a9e021e7fa1830: Bootstrap starting.
I20260926 01:12:32.140198 14752 tablet_bootstrap.cc:654] T db9d461f21454562ac5f5d627b91a45e P b193498b4e1a4ca091a9e021e7fa1830: Neither blocks nor log segments found. Creating new log.
I20260926 01:12:32.140695 14891 tablet_bootstrap.cc:492] T db9d461f21454562ac5f5d627b91a45e P cb9887afb40444c2b400da5aeff93486: Bootstrap starting.
I20260926 01:12:32.140995 14891 tablet_bootstrap.cc:654] T db9d461f21454562ac5f5d627b91a45e P cb9887afb40444c2b400da5aeff93486: Neither blocks nor log segments found. Creating new log.
I20260926 01:12:32.141052 14752 tablet_bootstrap.cc:492] T db9d461f21454562ac5f5d627b91a45e P b193498b4e1a4ca091a9e021e7fa1830: No bootstrap required, opened a new log
I20260926 01:12:32.141098 14752 ts_tablet_manager.cc:1397] T db9d461f21454562ac5f5d627b91a45e P b193498b4e1a4ca091a9e021e7fa1830: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 01:12:32.141263 14752 raft_consensus.cc:348] T db9d461f21454562ac5f5d627b91a45e P b193498b4e1a4ca091a9e021e7fa1830 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e3d9f9c88ff4617b5314f95cb3b5d47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45841 } } peers { permanent_uuid: "b193498b4e1a4ca091a9e021e7fa1830" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40439 } } peers { permanent_uuid: "cb9887afb40444c2b400da5aeff93486" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40171 } }
I20260926 01:12:32.141342 14752 raft_consensus.cc:374] T db9d461f21454562ac5f5d627b91a45e P b193498b4e1a4ca091a9e021e7fa1830 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:12:32.141388 14752 raft_consensus.cc:729] T db9d461f21454562ac5f5d627b91a45e P b193498b4e1a4ca091a9e021e7fa1830 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b193498b4e1a4ca091a9e021e7fa1830, State: Initialized, Role: FOLLOWER
I20260926 01:12:32.141441 14752 consensus_queue.cc:260] T db9d461f21454562ac5f5d627b91a45e P b193498b4e1a4ca091a9e021e7fa1830 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e3d9f9c88ff4617b5314f95cb3b5d47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45841 } } peers { permanent_uuid: "b193498b4e1a4ca091a9e021e7fa1830" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40439 } } peers { permanent_uuid: "cb9887afb40444c2b400da5aeff93486" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40171 } }
I20260926 01:12:32.141541 14752 ts_tablet_manager.cc:1428] T db9d461f21454562ac5f5d627b91a45e P b193498b4e1a4ca091a9e021e7fa1830: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 01:12:32.141605 14891 tablet_bootstrap.cc:492] T db9d461f21454562ac5f5d627b91a45e P cb9887afb40444c2b400da5aeff93486: No bootstrap required, opened a new log
I20260926 01:12:32.141649 14891 ts_tablet_manager.cc:1397] T db9d461f21454562ac5f5d627b91a45e P cb9887afb40444c2b400da5aeff93486: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260926 01:12:32.142030 14891 raft_consensus.cc:348] T db9d461f21454562ac5f5d627b91a45e P cb9887afb40444c2b400da5aeff93486 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e3d9f9c88ff4617b5314f95cb3b5d47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45841 } } peers { permanent_uuid: "b193498b4e1a4ca091a9e021e7fa1830" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40439 } } peers { permanent_uuid: "cb9887afb40444c2b400da5aeff93486" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40171 } }
I20260926 01:12:32.142099 14891 raft_consensus.cc:374] T db9d461f21454562ac5f5d627b91a45e P cb9887afb40444c2b400da5aeff93486 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:12:32.142130 14891 raft_consensus.cc:729] T db9d461f21454562ac5f5d627b91a45e P cb9887afb40444c2b400da5aeff93486 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb9887afb40444c2b400da5aeff93486, State: Initialized, Role: FOLLOWER
I20260926 01:12:32.142174 14891 consensus_queue.cc:260] T db9d461f21454562ac5f5d627b91a45e P cb9887afb40444c2b400da5aeff93486 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e3d9f9c88ff4617b5314f95cb3b5d47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45841 } } peers { permanent_uuid: "b193498b4e1a4ca091a9e021e7fa1830" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40439 } } peers { permanent_uuid: "cb9887afb40444c2b400da5aeff93486" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40171 } }
I20260926 01:12:32.142258 14891 ts_tablet_manager.cc:1428] T db9d461f21454562ac5f5d627b91a45e P cb9887afb40444c2b400da5aeff93486: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260926 01:12:32.143821 14867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:32.137746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45841 (local address 127.0.0.1:38454)
0926 01:12:32.137803 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0926 01:12:32.137840 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 01:12:32.137912 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:12:32.140348 (+ 2436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:12:32.140349 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:12:32.140352 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:12:32.140403 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:12:32.140405 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:12:32.141466 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:12:32.141468 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:12:32.142675 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:12:32.142678 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:12:32.143540 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:12:32.143541 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:12:32.143612 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:12:32.143639 (+ 27us) client_negotiation.cc:769] Sending connection context
0926 01:12:32.143690 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 01:12:32.143742 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260926 01:12:32.143841 15141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:32.139847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38454 (local address 127.0.98.3:45841)
0926 01:12:32.140179 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0926 01:12:32.140184 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 01:12:32.140196 (+ 12us) server_negotiation.cc:407] Connection header received
0926 01:12:32.140233 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:12:32.140234 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:12:32.140249 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:12:32.140326 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:12:32.140464 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:12:32.141315 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:12:32.142762 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:12:32.143455 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:12:32.143518 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:12:32.143583 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 01:12:32.143719 (+ 136us) server_negotiation.cc:299] Negotiation successful
0926 01:12:32.143762 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":118,"threads_started":1}
I20260926 01:12:32.144537 15069 tablet_service.cc:1467] Processing CreateTablet for tablet db9d461f21454562ac5f5d627b91a45e (DEFAULT_TABLE table=test-table [id=4d39f95b7200452b8ce81c66d560a687]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:12:32.144915 15069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db9d461f21454562ac5f5d627b91a45e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:12:32.147675 15144 tablet_bootstrap.cc:492] T db9d461f21454562ac5f5d627b91a45e P 5e3d9f9c88ff4617b5314f95cb3b5d47: Bootstrap starting.
I20260926 01:12:32.148277 15144 tablet_bootstrap.cc:654] T db9d461f21454562ac5f5d627b91a45e P 5e3d9f9c88ff4617b5314f95cb3b5d47: Neither blocks nor log segments found. Creating new log.
I20260926 01:12:32.148779 15144 log.cc:824] T db9d461f21454562ac5f5d627b91a45e P 5e3d9f9c88ff4617b5314f95cb3b5d47: Log is configured to *not* fsync() on all Append() calls
I20260926 01:12:32.149598 15144 tablet_bootstrap.cc:492] T db9d461f21454562ac5f5d627b91a45e P 5e3d9f9c88ff4617b5314f95cb3b5d47: No bootstrap required, opened a new log
I20260926 01:12:32.149664 15144 ts_tablet_manager.cc:1397] T db9d461f21454562ac5f5d627b91a45e P 5e3d9f9c88ff4617b5314f95cb3b5d47: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 01:12:32.151119 15144 raft_consensus.cc:348] T db9d461f21454562ac5f5d627b91a45e P 5e3d9f9c88ff4617b5314f95cb3b5d47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e3d9f9c88ff4617b5314f95cb3b5d47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45841 } } peers { permanent_uuid: "b193498b4e1a4ca091a9e021e7fa1830" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40439 } } peers { permanent_uuid: "cb9887afb40444c2b400da5aeff93486" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40171 } }
I20260926 01:12:32.151270 15144 raft_consensus.cc:374] T db9d461f21454562ac5f5d627b91a45e P 5e3d9f9c88ff4617b5314f95cb3b5d47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:12:32.151304 15144 raft_consensus.cc:729] T db9d461f21454562ac5f5d627b91a45e P 5e3d9f9c88ff4617b5314f95cb3b5d47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e3d9f9c88ff4617b5314f95cb3b5d47, State: Initialized, Role: FOLLOWER
I20260926 01:12:32.151435 15144 consensus_queue.cc:260] T db9d461f21454562ac5f5d627b91a45e P 5e3d9f9c88ff4617b5314f95cb3b5d47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e3d9f9c88ff4617b5314f95cb3b5d47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45841 } } peers { permanent_uuid: "b193498b4e1a4ca091a9e021e7fa1830" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40439 } } peers { permanent_uuid: "cb9887afb40444c2b400da5aeff93486" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40171 } }
I20260926 01:12:32.151691 15144 ts_tablet_manager.cc:1428] T db9d461f21454562ac5f5d627b91a45e P 5e3d9f9c88ff4617b5314f95cb3b5d47: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 01:12:32.151748 15135 heartbeater.cc:503] Master 127.0.98.62:32949 was elected leader, sending a full tablet report...
I20260926 01:12:32.197502 15146 raft_consensus.cc:482] T db9d461f21454562ac5f5d627b91a45e P 5e3d9f9c88ff4617b5314f95cb3b5d47 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:12:32.197636 15146 raft_consensus.cc:504] T db9d461f21454562ac5f5d627b91a45e P 5e3d9f9c88ff4617b5314f95cb3b5d47 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e3d9f9c88ff4617b5314f95cb3b5d47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45841 } } peers { permanent_uuid: "b193498b4e1a4ca091a9e021e7fa1830" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40439 } } peers { permanent_uuid: "cb9887afb40444c2b400da5aeff93486" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40171 } }
I20260926 01:12:32.198014 15146 leader_election.cc:288] T db9d461f21454562ac5f5d627b91a45e P 5e3d9f9c88ff4617b5314f95cb3b5d47 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b193498b4e1a4ca091a9e021e7fa1830 (127.0.98.1:40439), cb9887afb40444c2b400da5aeff93486 (127.0.98.2:40171)
I20260926 01:12:32.201584 15005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:32.198319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37303 (local address 127.0.98.2:40171)
0926 01:12:32.198360 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 01:12:32.198362 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:12:32.198369 (+ 7us) server_negotiation.cc:407] Connection header received
0926 01:12:32.198422 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:12:32.198423 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:12:32.198451 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:12:32.198533 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:12:32.198724 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:12:32.199309 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:12:32.200805 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:12:32.201237 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:12:32.201300 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:12:32.201385 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 01:12:32.201462 (+ 77us) server_negotiation.cc:299] Negotiation successful
0926 01:12:32.201503 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 01:12:32.201613 15137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:32.198213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40171 (local address 127.0.98.3:37303)
0926 01:12:32.198258 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0926 01:12:32.198271 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 01:12:32.198319 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:12:32.198554 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:12:32.198555 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:12:32.198557 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:12:32.198633 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:12:32.198636 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:12:32.199390 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:12:32.199391 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:12:32.200678 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:12:32.200682 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:12:32.201318 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:12:32.201319 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:12:32.201398 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:12:32.201402 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 01:12:32.201457 (+ 55us) client_negotiation.cc:240] Negotiation successful
0926 01:12:32.201510 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26,"mutex_wait_us":76}
I20260926 01:12:32.201622 14868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:32.198283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55115 (local address 127.0.98.1:40439)
0926 01:12:32.198334 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 01:12:32.198336 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:12:32.198551 (+ 215us) server_negotiation.cc:407] Connection header received
0926 01:12:32.198584 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:12:32.198585 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:12:32.198613 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:12:32.198684 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:12:32.198863 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:12:32.199344 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:12:32.200791 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:12:32.201263 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:12:32.201345 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:12:32.201431 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 01:12:32.201505 (+ 74us) server_negotiation.cc:299] Negotiation successful
0926 01:12:32.201548 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 01:12:32.201869 15147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:32.198216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40439 (local address 127.0.98.3:55115)
0926 01:12:32.198472 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0926 01:12:32.198482 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 01:12:32.198539 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:12:32.198716 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:12:32.198717 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:12:32.198720 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:12:32.198791 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:12:32.198792 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:12:32.199421 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:12:32.199422 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:12:32.200679 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:12:32.200682 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:12:32.201365 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:12:32.201367 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:12:32.201442 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:12:32.201447 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 01:12:32.201500 (+ 53us) client_negotiation.cc:240] Negotiation successful
0926 01:12:32.201547 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"mutex_wait_us":67,"thread_start_us":122,"threads_started":1}
I20260926 01:12:32.201958 14953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db9d461f21454562ac5f5d627b91a45e" candidate_uuid: "5e3d9f9c88ff4617b5314f95cb3b5d47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb9887afb40444c2b400da5aeff93486" is_pre_election: true
I20260926 01:12:32.202142 14953 raft_consensus.cc:2454] T db9d461f21454562ac5f5d627b91a45e P cb9887afb40444c2b400da5aeff93486 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e3d9f9c88ff4617b5314f95cb3b5d47 in term 0.
I20260926 01:12:32.202188 14814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db9d461f21454562ac5f5d627b91a45e" candidate_uuid: "5e3d9f9c88ff4617b5314f95cb3b5d47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b193498b4e1a4ca091a9e021e7fa1830" is_pre_election: true
I20260926 01:12:32.202347 14814 raft_consensus.cc:2454] T db9d461f21454562ac5f5d627b91a45e P b193498b4e1a4ca091a9e021e7fa1830 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e3d9f9c88ff4617b5314f95cb3b5d47 in term 0.
I20260926 01:12:32.202409 15024 leader_election.cc:302] T db9d461f21454562ac5f5d627b91a45e P 5e3d9f9c88ff4617b5314f95cb3b5d47 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e3d9f9c88ff4617b5314f95cb3b5d47, cb9887afb40444c2b400da5aeff93486; no voters:
I20260926 01:12:32.202551 15146 raft_consensus.cc:2781] T db9d461f21454562ac5f5d627b91a45e P 5e3d9f9c88ff4617b5314f95cb3b5d47 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:12:32.202602 15146 raft_consensus.cc:482] T db9d461f21454562ac5f5d627b91a45e P 5e3d9f9c88ff4617b5314f95cb3b5d47 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:12:32.202625 15146 raft_consensus.cc:3037] T db9d461f21454562ac5f5d627b91a45e P 5e3d9f9c88ff4617b5314f95cb3b5d47 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:12:32.203344 15146 raft_consensus.cc:504] T db9d461f21454562ac5f5d627b91a45e P 5e3d9f9c88ff4617b5314f95cb3b5d47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e3d9f9c88ff4617b5314f95cb3b5d47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45841 } } peers { permanent_uuid: "b193498b4e1a4ca091a9e021e7fa1830" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40439 } } peers { permanent_uuid: "cb9887afb40444c2b400da5aeff93486" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40171 } }
I20260926 01:12:32.203475 15146 leader_election.cc:288] T db9d461f21454562ac5f5d627b91a45e P 5e3d9f9c88ff4617b5314f95cb3b5d47 [CANDIDATE]: Term 1 election: Requested vote from peers b193498b4e1a4ca091a9e021e7fa1830 (127.0.98.1:40439), cb9887afb40444c2b400da5aeff93486 (127.0.98.2:40171)
I20260926 01:12:32.203639 14953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db9d461f21454562ac5f5d627b91a45e" candidate_uuid: "5e3d9f9c88ff4617b5314f95cb3b5d47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb9887afb40444c2b400da5aeff93486"
I20260926 01:12:32.203641 14814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db9d461f21454562ac5f5d627b91a45e" candidate_uuid: "5e3d9f9c88ff4617b5314f95cb3b5d47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b193498b4e1a4ca091a9e021e7fa1830"
I20260926 01:12:32.203716 14814 raft_consensus.cc:3037] T db9d461f21454562ac5f5d627b91a45e P b193498b4e1a4ca091a9e021e7fa1830 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:12:32.203717 14953 raft_consensus.cc:3037] T db9d461f21454562ac5f5d627b91a45e P cb9887afb40444c2b400da5aeff93486 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:12:32.204391 14953 raft_consensus.cc:2454] T db9d461f21454562ac5f5d627b91a45e P cb9887afb40444c2b400da5aeff93486 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e3d9f9c88ff4617b5314f95cb3b5d47 in term 1.
I20260926 01:12:32.204463 14814 raft_consensus.cc:2454] T db9d461f21454562ac5f5d627b91a45e P b193498b4e1a4ca091a9e021e7fa1830 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e3d9f9c88ff4617b5314f95cb3b5d47 in term 1.
I20260926 01:12:32.204564 15024 leader_election.cc:302] T db9d461f21454562ac5f5d627b91a45e P 5e3d9f9c88ff4617b5314f95cb3b5d47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e3d9f9c88ff4617b5314f95cb3b5d47, cb9887afb40444c2b400da5aeff93486; no voters:
I20260926 01:12:32.204648 15146 raft_consensus.cc:2781] T db9d461f21454562ac5f5d627b91a45e P 5e3d9f9c88ff4617b5314f95cb3b5d47 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:12:32.204831 15146 raft_consensus.cc:686] T db9d461f21454562ac5f5d627b91a45e P 5e3d9f9c88ff4617b5314f95cb3b5d47 [term 1 LEADER]: Becoming Leader. State: Replica: 5e3d9f9c88ff4617b5314f95cb3b5d47, State: Running, Role: LEADER
I20260926 01:12:32.204944 15146 consensus_queue.cc:237] T db9d461f21454562ac5f5d627b91a45e P 5e3d9f9c88ff4617b5314f95cb3b5d47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e3d9f9c88ff4617b5314f95cb3b5d47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45841 } } peers { permanent_uuid: "b193498b4e1a4ca091a9e021e7fa1830" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40439 } } peers { permanent_uuid: "cb9887afb40444c2b400da5aeff93486" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40171 } }
I20260926 01:12:32.205983 14672 catalog_manager.cc:5495] T db9d461f21454562ac5f5d627b91a45e P 5e3d9f9c88ff4617b5314f95cb3b5d47 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e3d9f9c88ff4617b5314f95cb3b5d47 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "5e3d9f9c88ff4617b5314f95cb3b5d47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e3d9f9c88ff4617b5314f95cb3b5d47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45841 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b193498b4e1a4ca091a9e021e7fa1830" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb9887afb40444c2b400da5aeff93486" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40171 } health_report { overall_health: UNKNOWN } } }
I20260926 01:12:32.219386 15141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:32.216988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38458 (local address 127.0.98.3:45841)
0926 01:12:32.217072 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0926 01:12:32.217074 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:12:32.217084 (+ 10us) server_negotiation.cc:407] Connection header received
0926 01:12:32.217165 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:12:32.217166 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:12:32.217186 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:12:32.217248 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:12:32.217363 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:12:32.217919 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:12:32.218657 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:12:32.218847 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:12:32.218903 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:12:32.219051 (+ 148us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:12:32.219161 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:12:32.219235 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 01:12:32.219255 (+ 20us) server_negotiation.cc:299] Negotiation successful
0926 01:12:32.219279 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 01:12:32.221441 14814 raft_consensus.cc:1261] T db9d461f21454562ac5f5d627b91a45e P b193498b4e1a4ca091a9e021e7fa1830 [term 1 FOLLOWER]: Refusing update from remote peer 5e3d9f9c88ff4617b5314f95cb3b5d47: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:12:32.221534 14953 raft_consensus.cc:1261] T db9d461f21454562ac5f5d627b91a45e P cb9887afb40444c2b400da5aeff93486 [term 1 FOLLOWER]: Refusing update from remote peer 5e3d9f9c88ff4617b5314f95cb3b5d47: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:12:32.221848 15148 consensus_queue.cc:1035] T db9d461f21454562ac5f5d627b91a45e P 5e3d9f9c88ff4617b5314f95cb3b5d47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b193498b4e1a4ca091a9e021e7fa1830" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:12:32.221993 15146 consensus_queue.cc:1035] T db9d461f21454562ac5f5d627b91a45e P 5e3d9f9c88ff4617b5314f95cb3b5d47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb9887afb40444c2b400da5aeff93486" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:12:32.224141 15150 mvcc.cc:205] Tried to move back new op lower bound from 7333417583495745536 to 7333417583432192000. Current Snapshot: MvccSnapshot[applied={T|T < 7333417583495745536}]
I20260926 01:12:32.224622 15151 mvcc.cc:205] Tried to move back new op lower bound from 7333417583495745536 to 7333417583432192000. Current Snapshot: MvccSnapshot[applied={T|T < 7333417583495745536}]
I20260926 01:12:32.225596 15153 mvcc.cc:205] Tried to move back new op lower bound from 7333417583495745536 to 7333417583432192000. Current Snapshot: MvccSnapshot[applied={T|T < 7333417583495745536}]
I20260926 01:12:32.227368 14670 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:59340:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:12:32.227466 14670 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:59340:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:12:32.228209 14670 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 396a0b42495f4fd9a266d7c98fa4012f: Sending DeleteTablet for 3 replicas of tablet db9d461f21454562ac5f5d627b91a45e
I20260926 01:12:32.228684 14794 tablet_service.cc:1514] Processing DeleteTablet for tablet db9d461f21454562ac5f5d627b91a45e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:12:32 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34440
I20260926 01:12:32.228698 14933 tablet_service.cc:1514] Processing DeleteTablet for tablet db9d461f21454562ac5f5d627b91a45e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:12:32 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33116
I20260926 01:12:32.228790 14869 tablet_replica.cc:330] T db9d461f21454562ac5f5d627b91a45e P b193498b4e1a4ca091a9e021e7fa1830: stopping tablet replica
I20260926 01:12:32.228834 14869 raft_consensus.cc:2229] T db9d461f21454562ac5f5d627b91a45e P b193498b4e1a4ca091a9e021e7fa1830 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:12:32.228816 15006 tablet_replica.cc:330] T db9d461f21454562ac5f5d627b91a45e P cb9887afb40444c2b400da5aeff93486: stopping tablet replica
I20260926 01:12:32.228884 14869 raft_consensus.cc:2258] T db9d461f21454562ac5f5d627b91a45e P b193498b4e1a4ca091a9e021e7fa1830 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:12:32.228859 15069 tablet_service.cc:1514] Processing DeleteTablet for tablet db9d461f21454562ac5f5d627b91a45e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:12:32 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38454
I20260926 01:12:32.228904 15006 raft_consensus.cc:2229] T db9d461f21454562ac5f5d627b91a45e P cb9887afb40444c2b400da5aeff93486 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:12:32.228965 15006 raft_consensus.cc:2258] T db9d461f21454562ac5f5d627b91a45e P cb9887afb40444c2b400da5aeff93486 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:12:32.229025 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 14710
I20260926 01:12:32.229300 15006 ts_tablet_manager.cc:1905] T db9d461f21454562ac5f5d627b91a45e P cb9887afb40444c2b400da5aeff93486: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:12:32.230084 15159 tablet_replica.cc:330] T db9d461f21454562ac5f5d627b91a45e P 5e3d9f9c88ff4617b5314f95cb3b5d47: stopping tablet replica
I20260926 01:12:32.230204 15159 raft_consensus.cc:2229] T db9d461f21454562ac5f5d627b91a45e P 5e3d9f9c88ff4617b5314f95cb3b5d47 [term 1 LEADER]: Raft consensus shutting down.
I20260926 01:12:32.230314 15159 raft_consensus.cc:2258] T db9d461f21454562ac5f5d627b91a45e P 5e3d9f9c88ff4617b5314f95cb3b5d47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:12:32.230942 15006 ts_tablet_manager.cc:1918] T db9d461f21454562ac5f5d627b91a45e P cb9887afb40444c2b400da5aeff93486: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:12:32.230997 15006 log.cc:1196] T db9d461f21454562ac5f5d627b91a45e P cb9887afb40444c2b400da5aeff93486: Deleting WAL directory at /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-1/wal/wals/db9d461f21454562ac5f5d627b91a45e
I20260926 01:12:32.231155 15006 ts_tablet_manager.cc:1939] T db9d461f21454562ac5f5d627b91a45e P cb9887afb40444c2b400da5aeff93486: Deleting consensus metadata
I20260926 01:12:32.234865 14658 catalog_manager.cc:4841] TS cb9887afb40444c2b400da5aeff93486 (127.0.98.2:40171): tablet db9d461f21454562ac5f5d627b91a45e (table test-table [id=4d39f95b7200452b8ce81c66d560a687]) successfully deleted
W20260926 01:12:32.235790 14657 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:40439 (error 108)
I20260926 01:12:32.236124 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 14862
I20260926 01:12:32.236482 15159 ts_tablet_manager.cc:1905] T db9d461f21454562ac5f5d627b91a45e P 5e3d9f9c88ff4617b5314f95cb3b5d47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:12:32.238230 15159 ts_tablet_manager.cc:1918] T db9d461f21454562ac5f5d627b91a45e P 5e3d9f9c88ff4617b5314f95cb3b5d47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:12:32.238309 15159 log.cc:1196] T db9d461f21454562ac5f5d627b91a45e P 5e3d9f9c88ff4617b5314f95cb3b5d47: Deleting WAL directory at /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-2/wal/wals/db9d461f21454562ac5f5d627b91a45e
I20260926 01:12:32.238583 15159 ts_tablet_manager.cc:1939] T db9d461f21454562ac5f5d627b91a45e P 5e3d9f9c88ff4617b5314f95cb3b5d47: Deleting consensus metadata
I20260926 01:12:32.238976 14867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:32.238825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40439 (local address 127.0.0.1:34444)
0926 01:12:32.238878 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0926 01:12:32.238936 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260926 01:12:32.239109 14657 catalog_manager.cc:4584] TS b193498b4e1a4ca091a9e021e7fa1830 (127.0.98.1:40439): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet db9d461f21454562ac5f5d627b91a45e: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40439: connect: Connection refused (error 111)
I20260926 01:12:32.239681 14658 catalog_manager.cc:4841] TS 5e3d9f9c88ff4617b5314f95cb3b5d47 (127.0.98.3:45841): tablet db9d461f21454562ac5f5d627b91a45e (table test-table [id=4d39f95b7200452b8ce81c66d560a687]) successfully deleted
I20260926 01:12:32.243208 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 15004
I20260926 01:12:32.248076 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 14639
I20260926 01:12:32.252681 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32949
--webserver_interface=127.0.98.62
--webserver_port=46799
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:32949
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:12:32.336688 15161 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:12:32.336932 15161 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:12:32.336957 15161 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:12:32.338311 15161 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:12:32.338362 15161 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:12:32.338376 15161 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:12:32.338387 15161 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:12:32.338398 15161 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:12:32.340059 15161 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:32949
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32949
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=46799
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15161
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:12:32.340323 15161 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:12:32.340564 15161 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:12:32.345515 15169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:12:32.345538 15167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:12:32.345538 15166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:12:32.345852 15161 server_base.cc:1034] running on GCE node
I20260926 01:12:32.346118 15161 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:12:32.346554 15161 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:12:32.347838 15161 hybrid_clock.cc:648] HybridClock initialized: now 1790385152347798 us; error 47 us; skew 500 ppm
Sep 26 01:12:32 dist-test-slave-7b22 krb5kdc[14095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385152, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:12:32.350939 15161 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 01:12:32.351452 15161 webserver.cc:460] Webserver started at http://127.0.98.62:46799/ using document root <none> and password file <none>
I20260926 01:12:32.351642 15161 fs_manager.cc:359] Metadata directory not provided
I20260926 01:12:32.351684 15161 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:12:32.353106 15161 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:12:32.353842 15176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:12:32.354028 15161 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:12:32.354120 15161 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/master-0/wal
uuid: "396a0b42495f4fd9a266d7c98fa4012f"
format_stamp: "Formatted at 2026-09-26 01:12:30 on dist-test-slave-7b22"
server_key: "34eef3d71352829a61341ec88d1333f2"
server_key_iv: "a3904234ff1334e45601d2cf3db600ba"
server_key_version: "encryptionkey@0"
I20260926 01:12:32.354521 15161 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:12:32.383355 15161 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:12:32.385926 15161 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:12:32.386056 15161 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:12:32.390123 15161 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:32949
I20260926 01:12:32.390167 15228 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:32949 every 8 connection(s)
I20260926 01:12:32.390553 15161 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
I20260926 01:12:32.392123 15229 sys_catalog.cc:263] Verifying existing consensus state
I20260926 01:12:32.392814 15229 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 396a0b42495f4fd9a266d7c98fa4012f: Bootstrap starting.
I20260926 01:12:32.398996 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 15161
I20260926 01:12:32.399325 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:40439
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:32949
--webserver_port=43983
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 01:12:32.401924 15229 log.cc:824] T 00000000000000000000000000000000 P 396a0b42495f4fd9a266d7c98fa4012f: Log is configured to *not* fsync() on all Append() calls
I20260926 01:12:32.404179 15229 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 396a0b42495f4fd9a266d7c98fa4012f: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 01:12:32.404451 15229 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 396a0b42495f4fd9a266d7c98fa4012f: Bootstrap complete.
I20260926 01:12:32.406608 15229 raft_consensus.cc:348] T 00000000000000000000000000000000 P 396a0b42495f4fd9a266d7c98fa4012f [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "396a0b42495f4fd9a266d7c98fa4012f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32949 } }
I20260926 01:12:32.406834 15229 raft_consensus.cc:729] T 00000000000000000000000000000000 P 396a0b42495f4fd9a266d7c98fa4012f [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 396a0b42495f4fd9a266d7c98fa4012f, State: Initialized, Role: FOLLOWER
I20260926 01:12:32.406993 15229 consensus_queue.cc:260] T 00000000000000000000000000000000 P 396a0b42495f4fd9a266d7c98fa4012f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "396a0b42495f4fd9a266d7c98fa4012f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32949 } }
I20260926 01:12:32.407068 15229 raft_consensus.cc:388] T 00000000000000000000000000000000 P 396a0b42495f4fd9a266d7c98fa4012f [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:12:32.407115 15229 raft_consensus.cc:482] T 00000000000000000000000000000000 P 396a0b42495f4fd9a266d7c98fa4012f [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:12:32.407163 15229 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 396a0b42495f4fd9a266d7c98fa4012f [term 2 FOLLOWER]: Advancing to term 3
I20260926 01:12:32.408170 15229 raft_consensus.cc:504] T 00000000000000000000000000000000 P 396a0b42495f4fd9a266d7c98fa4012f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "396a0b42495f4fd9a266d7c98fa4012f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32949 } }
I20260926 01:12:32.408269 15229 leader_election.cc:302] T 00000000000000000000000000000000 P 396a0b42495f4fd9a266d7c98fa4012f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 396a0b42495f4fd9a266d7c98fa4012f; no voters:
I20260926 01:12:32.408435 15229 leader_election.cc:288] T 00000000000000000000000000000000 P 396a0b42495f4fd9a266d7c98fa4012f [CANDIDATE]: Term 3 election: Requested vote from peers
I20260926 01:12:32.408522 15233 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 396a0b42495f4fd9a266d7c98fa4012f [term 3 FOLLOWER]: Leader election won for term 3
I20260926 01:12:32.408675 15233 raft_consensus.cc:686] T 00000000000000000000000000000000 P 396a0b42495f4fd9a266d7c98fa4012f [term 3 LEADER]: Becoming Leader. State: Replica: 396a0b42495f4fd9a266d7c98fa4012f, State: Running, Role: LEADER
I20260926 01:12:32.408780 15229 sys_catalog.cc:564] T 00000000000000000000000000000000 P 396a0b42495f4fd9a266d7c98fa4012f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:12:32.408771 15233 consensus_queue.cc:237] T 00000000000000000000000000000000 P 396a0b42495f4fd9a266d7c98fa4012f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "396a0b42495f4fd9a266d7c98fa4012f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32949 } }
I20260926 01:12:32.409087 15234 sys_catalog.cc:455] T 00000000000000000000000000000000 P 396a0b42495f4fd9a266d7c98fa4012f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "396a0b42495f4fd9a266d7c98fa4012f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "396a0b42495f4fd9a266d7c98fa4012f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32949 } } }
I20260926 01:12:32.409159 15234 sys_catalog.cc:458] T 00000000000000000000000000000000 P 396a0b42495f4fd9a266d7c98fa4012f [sys.catalog]: This master's current role is: LEADER
I20260926 01:12:32.409288 15235 sys_catalog.cc:455] T 00000000000000000000000000000000 P 396a0b42495f4fd9a266d7c98fa4012f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 396a0b42495f4fd9a266d7c98fa4012f. Latest consensus state: current_term: 3 leader_uuid: "396a0b42495f4fd9a266d7c98fa4012f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "396a0b42495f4fd9a266d7c98fa4012f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32949 } } }
I20260926 01:12:32.409332 15235 sys_catalog.cc:458] T 00000000000000000000000000000000 P 396a0b42495f4fd9a266d7c98fa4012f [sys.catalog]: This master's current role is: LEADER
I20260926 01:12:32.409660 15238 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:12:32.410490 15238 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:12:32.410593 15238 catalog_manager.cc:1259] Loaded cluster ID: 9c532ce79c0e4442ad110b86d288c165
I20260926 01:12:32.410841 15238 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260926 01:12:32.411108 15238 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f3b6e6d19ad google::LogMessage::Fail() at ??:0
@ 0x7f3b6e6d488a google::LogMessage::SendToLog() at ??:0
@ 0x7f3b6e6d14de google::LogMessage::Flush() at ??:0
@ 0x7f3b6e6d2a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f3b765f20f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f3b765fb8b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f3b7059f0d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f3b70594954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f3b713f86db start_thread at ??:0
@ 0x7f3b6d1cf61f clone at ??:0
W20260926 01:12:32.493402 15231 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:12:32.493621 15231 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:12:32.493647 15231 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:12:32.494980 15231 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:12:32.495044 15231 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:12:32.495085 15231 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 01:12:32.497501 15231 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:40439
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=43983
--tserver_master_addrs=127.0.98.62:32949
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15231
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:12:32.497797 15231 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:12:32.498104 15231 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:12:32.503592 15265 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:12:32.503649 15266 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:12:32.503789 15268 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:12:32.504487 15231 server_base.cc:1034] running on GCE node
I20260926 01:12:32.504721 15231 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:12:32.505005 15231 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:12:32.506157 15231 hybrid_clock.cc:648] HybridClock initialized: now 1790385152506126 us; error 45 us; skew 500 ppm
Sep 26 01:12:32 dist-test-slave-7b22 krb5kdc[14095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385152, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:12:32.509250 15231 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 01:12:32.510052 15231 webserver.cc:460] Webserver started at http://127.0.98.1:43983/ using document root <none> and password file <none>
I20260926 01:12:32.510419 15231 fs_manager.cc:359] Metadata directory not provided
I20260926 01:12:32.510514 15231 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:12:32.512458 15231 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 01:12:32.513389 15275 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:12:32.513638 15231 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:12:32.513737 15231 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-0/wal
uuid: "b193498b4e1a4ca091a9e021e7fa1830"
format_stamp: "Formatted at 2026-09-26 01:12:31 on dist-test-slave-7b22"
server_key: "fdd3b728c1189ff7ea60b8bc9bd976b7"
server_key_iv: "c2490517d7a60083a59be57bc4e4ed69"
server_key_version: "encryptionkey@0"
I20260926 01:12:32.514187 15231 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:12:32.534417 15231 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:12:32.536566 15231 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:12:32.536713 15231 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:12:32.536949 15231 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:12:32.537372 15282 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 01:12:32.540026 15231 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 01:12:32.540099 15231 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.003s user 0.000s sys 0.000s
I20260926 01:12:32.540139 15231 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 01:12:32.540850 15231 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 01:12:32.540884 15231 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260926 01:12:32.540984 15282 tablet_bootstrap.cc:492] T db9d461f21454562ac5f5d627b91a45e P b193498b4e1a4ca091a9e021e7fa1830: Bootstrap starting.
I20260926 01:12:32.547930 15231 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40439
I20260926 01:12:32.548029 15389 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40439 every 8 connection(s)
I20260926 01:12:32.548349 15231 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-0/data/info.pb
I20260926 01:12:32.553161 15282 log.cc:824] T db9d461f21454562ac5f5d627b91a45e P b193498b4e1a4ca091a9e021e7fa1830: Log is configured to *not* fsync() on all Append() calls
I20260926 01:12:32.555387 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 15231
I20260926 01:12:32.555666 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:40171
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:32949
--webserver_port=37981
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 01:12:32.556069 15282 tablet_bootstrap.cc:492] T db9d461f21454562ac5f5d627b91a45e P b193498b4e1a4ca091a9e021e7fa1830: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 01:12:32.556341 15282 tablet_bootstrap.cc:492] T db9d461f21454562ac5f5d627b91a45e P b193498b4e1a4ca091a9e021e7fa1830: Bootstrap complete.
I20260926 01:12:32.557265 15282 ts_tablet_manager.cc:1397] T db9d461f21454562ac5f5d627b91a45e P b193498b4e1a4ca091a9e021e7fa1830: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.004s
I20260926 01:12:32.558806 15282 raft_consensus.cc:348] T db9d461f21454562ac5f5d627b91a45e P b193498b4e1a4ca091a9e021e7fa1830 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e3d9f9c88ff4617b5314f95cb3b5d47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45841 } } peers { permanent_uuid: "b193498b4e1a4ca091a9e021e7fa1830" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40439 } } peers { permanent_uuid: "cb9887afb40444c2b400da5aeff93486" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40171 } }
I20260926 01:12:32.559128 15282 raft_consensus.cc:729] T db9d461f21454562ac5f5d627b91a45e P b193498b4e1a4ca091a9e021e7fa1830 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b193498b4e1a4ca091a9e021e7fa1830, State: Initialized, Role: FOLLOWER
I20260926 01:12:32.559273 15282 consensus_queue.cc:260] T db9d461f21454562ac5f5d627b91a45e P b193498b4e1a4ca091a9e021e7fa1830 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e3d9f9c88ff4617b5314f95cb3b5d47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45841 } } peers { permanent_uuid: "b193498b4e1a4ca091a9e021e7fa1830" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40439 } } peers { permanent_uuid: "cb9887afb40444c2b400da5aeff93486" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40171 } }
I20260926 01:12:32.559593 15282 ts_tablet_manager.cc:1428] T db9d461f21454562ac5f5d627b91a45e P b193498b4e1a4ca091a9e021e7fa1830: Time spent starting tablet: real 0.002s user 0.002s sys 0.003s
W20260926 01:12:32.623374 15392 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:12:32.549064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32949 (local address 127.0.98.1:51535)
0926 01:12:32.549238 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0926 01:12:32.549284 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 01:12:32.549764 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:12:32.623185 (+ 73421us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32949: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":75,"threads_started":1}
W20260926 01:12:32.623682 15390 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:32949 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:32949: Client connection negotiation failed: client connection to 127.0.98.62:32949: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 01:12:32.624002 15392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:32.623892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32949 (local address 127.0.98.1:59705)
0926 01:12:32.623939 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0926 01:12:32.623981 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260926 01:12:32.624302 15392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:32.624220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32949 (local address 127.0.98.1:52185)
0926 01:12:32.624260 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0926 01:12:32.624283 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260926 01:12:32.624419 15390 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 01:12:32.643003 15394 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:12:32.643242 15394 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:12:32.643268 15394 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:12:32.644542 15394 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:12:32.644593 15394 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:12:32.644620 15394 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 01:12:32.646229 15394 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:40171
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=37981
--tserver_master_addrs=127.0.98.62:32949
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15394
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:12:32.646468 15394 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:12:32.646692 15394 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:12:32.651952 15403 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:12:32.651925 15400 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:12:32.652043 15401 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:12:32.652060 15394 server_base.cc:1034] running on GCE node
I20260926 01:12:32.652494 15394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:12:32.652737 15394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:12:32.653895 15394 hybrid_clock.cc:648] HybridClock initialized: now 1790385152653881 us; error 30 us; skew 500 ppm
Sep 26 01:12:32 dist-test-slave-7b22 krb5kdc[14095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385152, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:12:32.656427 15394 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 01:12:32.656876 15394 webserver.cc:460] Webserver started at http://127.0.98.2:37981/ using document root <none> and password file <none>
I20260926 01:12:32.657087 15394 fs_manager.cc:359] Metadata directory not provided
I20260926 01:12:32.657130 15394 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:12:32.658497 15394 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 01:12:32.659118 15410 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:12:32.659284 15394 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 01:12:32.659363 15394 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-1/wal
uuid: "cb9887afb40444c2b400da5aeff93486"
format_stamp: "Formatted at 2026-09-26 01:12:31 on dist-test-slave-7b22"
server_key: "9349ff8e45cff9d62e0eb3b58ca443e3"
server_key_iv: "73f73772a4aeec2ab5af356f748f85df"
server_key_version: "encryptionkey@0"
I20260926 01:12:32.659718 15394 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:12:32.670231 15394 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:12:32.672256 15394 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:12:32.672379 15394 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:12:32.672605 15394 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:12:32.672925 15394 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:12:32.672958 15394 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:12:32.672981 15394 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:12:32.672991 15394 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:12:32.678918 15394 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40171
I20260926 01:12:32.679122 15523 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40171 every 8 connection(s)
I20260926 01:12:32.679332 15394 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-1/data/info.pb
I20260926 01:12:32.680402 15526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:32.680027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32949 (local address 127.0.98.2:51947)
0926 01:12:32.680190 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 01:12:32.680264 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":86,"threads_started":1}
W20260926 01:12:32.680869 15524 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:32949 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:32949: Client connection negotiation failed: client connection to 127.0.98.62:32949: connect: Connection refused (error 111)
I20260926 01:12:32.681303 15526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:32.681185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32949 (local address 127.0.98.2:58629)
0926 01:12:32.681250 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0926 01:12:32.681280 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260926 01:12:32.681638 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 15394
I20260926 01:12:32.681758 15526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:32.681635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32949 (local address 127.0.98.2:47331)
0926 01:12:32.681696 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0926 01:12:32.681730 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260926 01:12:32.681876 15524 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 01:12:32.681911 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:45841
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:32949
--webserver_port=44729
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:12:32.765808 15527 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:12:32.766094 15527 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:12:32.766121 15527 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:12:32.767400 15527 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:12:32.767454 15527 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:12:32.767482 15527 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 01:12:32.769064 15527 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:45841
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=44729
--tserver_master_addrs=127.0.98.62:32949
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15527
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:12:32.769312 15527 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:12:32.769548 15527 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:12:32.775162 15533 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:12:32.775156 15535 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:12:32.775282 15527 server_base.cc:1034] running on GCE node
W20260926 01:12:32.775471 15532 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:12:32.775695 15527 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:12:32.775991 15527 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:12:32.777171 15527 hybrid_clock.cc:648] HybridClock initialized: now 1790385152777157 us; error 33 us; skew 500 ppm
Sep 26 01:12:32 dist-test-slave-7b22 krb5kdc[14095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385152, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:12:32.784003 15527 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 01:12:32.784677 15527 webserver.cc:460] Webserver started at http://127.0.98.3:44729/ using document root <none> and password file <none>
I20260926 01:12:32.784929 15527 fs_manager.cc:359] Metadata directory not provided
I20260926 01:12:32.784989 15527 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:12:32.786458 15527 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 01:12:32.787261 15542 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:12:32.787453 15527 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:12:32.787547 15527 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-2/wal
uuid: "5e3d9f9c88ff4617b5314f95cb3b5d47"
format_stamp: "Formatted at 2026-09-26 01:12:31 on dist-test-slave-7b22"
server_key: "8bfee5695dbe2c9914163b20a004de48"
server_key_iv: "1c2a4134373cdf7cf9808899feaa641f"
server_key_version: "encryptionkey@0"
I20260926 01:12:32.787911 15527 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:12:32.799516 15527 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:12:32.801966 15527 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:12:32.802127 15527 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:12:32.802378 15527 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:12:32.802696 15527 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:12:32.802733 15527 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:12:32.802767 15527 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:12:32.802794 15527 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:12:32.809407 15527 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45841
I20260926 01:12:32.809597 15655 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45841 every 8 connection(s)
I20260926 01:12:32.809959 15527 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384993574762-392-0/minicluster-data/ts-2/data/info.pb
I20260926 01:12:32.811273 15658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:32.810849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32949 (local address 127.0.98.3:57837)
0926 01:12:32.811045 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0926 01:12:32.811122 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":85,"threads_started":1}
W20260926 01:12:32.811699 15656 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:32949 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:32949: Client connection negotiation failed: client connection to 127.0.98.62:32949: connect: Connection refused (error 111)
I20260926 01:12:32.812002 15658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:32.811900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32949 (local address 127.0.98.3:58547)
0926 01:12:32.811950 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0926 01:12:32.811980 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260926 01:12:32.812351 15658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:32.812245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32949 (local address 127.0.98.3:49013)
0926 01:12:32.812302 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0926 01:12:32.812331 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260926 01:12:32.812556 15656 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 01:12:32.817348 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 15527
I20260926 01:12:33.625054 15660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:33.624769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32949 (local address 127.0.98.1:45411)
0926 01:12:33.624937 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 01:12:33.624975 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":62,"threads_started":1}
I20260926 01:12:33.682478 15661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:33.682234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32949 (local address 127.0.98.2:45327)
0926 01:12:33.682360 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0926 01:12:33.682399 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":62,"threads_started":1}
I20260926 01:12:33.813174 15662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:33.812929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32949 (local address 127.0.98.3:44503)
0926 01:12:33.813086 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 01:12:33.813117 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":82,"threads_started":1}
I20260926 01:12:33.861315 15663 raft_consensus.cc:482] T db9d461f21454562ac5f5d627b91a45e P b193498b4e1a4ca091a9e021e7fa1830 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:12:33.861454 15663 raft_consensus.cc:504] T db9d461f21454562ac5f5d627b91a45e P b193498b4e1a4ca091a9e021e7fa1830 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e3d9f9c88ff4617b5314f95cb3b5d47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45841 } } peers { permanent_uuid: "b193498b4e1a4ca091a9e021e7fa1830" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40439 } } peers { permanent_uuid: "cb9887afb40444c2b400da5aeff93486" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40171 } }
I20260926 01:12:33.861805 15663 leader_election.cc:288] T db9d461f21454562ac5f5d627b91a45e P b193498b4e1a4ca091a9e021e7fa1830 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5e3d9f9c88ff4617b5314f95cb3b5d47 (127.0.98.3:45841), cb9887afb40444c2b400da5aeff93486 (127.0.98.2:40171)
Sep 26 01:12:33 dist-test-slave-7b22 krb5kdc[14095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385152, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
Sep 26 01:12:33 dist-test-slave-7b22 krb5kdc[14095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385152, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260926 01:12:33.872438 15660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:33.862020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45841 (local address 127.0.98.1:43623)
0926 01:12:33.862078 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0926 01:12:33.862092 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 01:12:33.862162 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:12:33.863041 (+ 879us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:12:33.863044 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:12:33.863420 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:12:33.863610 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:12:33.863616 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:12:33.864755 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:12:33.864756 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:12:33.865477 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:12:33.865479 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:12:33.866158 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:12:33.866159 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:12:33.866263 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:12:33.866846 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:12:33.866851 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:12:33.868405 (+ 1554us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:12:33.870518 (+ 2113us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:12:33.870526 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:12:33.870529 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:12:33.870871 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:12:33.871198 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:12:33.871199 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:12:33.871199 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:12:33.871366 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:12:33.871868 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:12:33.871874 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:12:33.872031 (+ 157us) client_negotiation.cc:769] Sending connection context
0926 01:12:33.872139 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 01:12:33.872324 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":60}
I20260926 01:12:33.872454 15666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:33.862321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51847 (local address 127.0.98.2:40171)
0926 01:12:33.862480 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0926 01:12:33.862488 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 01:12:33.862512 (+ 24us) server_negotiation.cc:407] Connection header received
0926 01:12:33.862951 (+ 439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:12:33.862954 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:12:33.863043 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:12:33.863173 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:12:33.863715 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:12:33.864558 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:12:33.865557 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:12:33.866071 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:12:33.866173 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:12:33.868856 (+ 2683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:12:33.868861 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:12:33.868861 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:12:33.868873 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:12:33.870546 (+ 1673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:12:33.870989 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:12:33.870992 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:12:33.870992 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:12:33.871125 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:12:33.871459 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:12:33.871460 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:12:33.871460 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:12:33.871797 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:12:33.871883 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 01:12:33.872198 (+ 315us) server_negotiation.cc:299] Negotiation successful
0926 01:12:33.872337 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":70,"threads_started":1}
I20260926 01:12:33.872495 15664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:33.862136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40171 (local address 127.0.98.1:51847)
0926 01:12:33.862331 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0926 01:12:33.862341 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 01:12:33.862384 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:12:33.863161 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:12:33.863162 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:12:33.863418 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:12:33.863610 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:12:33.863616 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:12:33.864659 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:12:33.864660 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:12:33.865475 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:12:33.865478 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:12:33.866152 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:12:33.866153 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:12:33.866264 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:12:33.866796 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:12:33.866804 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:12:33.868754 (+ 1950us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:12:33.870612 (+ 1858us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:12:33.870612 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:12:33.870613 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:12:33.870871 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:12:33.871198 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:12:33.871199 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:12:33.871199 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:12:33.871365 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:12:33.871868 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:12:33.871874 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:12:33.872027 (+ 153us) client_negotiation.cc:769] Sending connection context
0926 01:12:33.872139 (+ 112us) client_negotiation.cc:240] Negotiation successful
0926 01:12:33.872387 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"mutex_wait_us":126,"thread_start_us":79,"threads_started":1}
I20260926 01:12:33.872458 15665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:33.862148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43623 (local address 127.0.98.3:45841)
0926 01:12:33.862309 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 01:12:33.862318 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 01:12:33.862344 (+ 26us) server_negotiation.cc:407] Connection header received
0926 01:12:33.862790 (+ 446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:12:33.862792 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:12:33.862849 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:12:33.862989 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:12:33.863725 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:12:33.864669 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:12:33.865576 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:12:33.866075 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:12:33.866173 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:12:33.868552 (+ 2379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:12:33.868557 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:12:33.868558 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:12:33.868571 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:12:33.870436 (+ 1865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:12:33.870988 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:12:33.870992 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:12:33.870992 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:12:33.871125 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:12:33.871459 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:12:33.871460 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:12:33.871460 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:12:33.871797 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:12:33.871883 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 01:12:33.872198 (+ 315us) server_negotiation.cc:299] Negotiation successful
0926 01:12:33.872337 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":63,"threads_started":1}
W20260926 01:12:33.873283 15278 leader_election.cc:341] T db9d461f21454562ac5f5d627b91a45e P b193498b4e1a4ca091a9e021e7fa1830 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cb9887afb40444c2b400da5aeff93486 (127.0.98.2:40171): Not found: Tablet not found: db9d461f21454562ac5f5d627b91a45e
W20260926 01:12:33.873406 15278 leader_election.cc:341] T db9d461f21454562ac5f5d627b91a45e P b193498b4e1a4ca091a9e021e7fa1830 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5e3d9f9c88ff4617b5314f95cb3b5d47 (127.0.98.3:45841): Not found: Tablet not found: db9d461f21454562ac5f5d627b91a45e
I20260926 01:12:33.873442 15278 leader_election.cc:302] T db9d461f21454562ac5f5d627b91a45e P b193498b4e1a4ca091a9e021e7fa1830 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b193498b4e1a4ca091a9e021e7fa1830; no voters: 5e3d9f9c88ff4617b5314f95cb3b5d47, cb9887afb40444c2b400da5aeff93486
I20260926 01:12:33.873575 15663 raft_consensus.cc:2726] T db9d461f21454562ac5f5d627b91a45e P b193498b4e1a4ca091a9e021e7fa1830 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 01:12:34.625960 15667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:34.625603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32949 (local address 127.0.98.1:58703)
0926 01:12:34.625756 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 01:12:34.625812 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":70,"threads_started":1}
I20260926 01:12:34.683216 15668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:34.682987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32949 (local address 127.0.98.2:52987)
0926 01:12:34.683128 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0926 01:12:34.683170 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":64,"threads_started":1}
I20260926 01:12:34.813912 15669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:34.813680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32949 (local address 127.0.98.3:53335)
0926 01:12:34.813799 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0926 01:12:34.813861 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":62,"threads_started":1}
I20260926 01:12:35.541566 15670 raft_consensus.cc:482] T db9d461f21454562ac5f5d627b91a45e P b193498b4e1a4ca091a9e021e7fa1830 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:12:35.541664 15670 raft_consensus.cc:504] T db9d461f21454562ac5f5d627b91a45e P b193498b4e1a4ca091a9e021e7fa1830 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e3d9f9c88ff4617b5314f95cb3b5d47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45841 } } peers { permanent_uuid: "b193498b4e1a4ca091a9e021e7fa1830" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40439 } } peers { permanent_uuid: "cb9887afb40444c2b400da5aeff93486" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40171 } }
I20260926 01:12:35.541787 15670 leader_election.cc:288] T db9d461f21454562ac5f5d627b91a45e P b193498b4e1a4ca091a9e021e7fa1830 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5e3d9f9c88ff4617b5314f95cb3b5d47 (127.0.98.3:45841), cb9887afb40444c2b400da5aeff93486 (127.0.98.2:40171)
W20260926 01:12:35.542219 15278 leader_election.cc:341] T db9d461f21454562ac5f5d627b91a45e P b193498b4e1a4ca091a9e021e7fa1830 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cb9887afb40444c2b400da5aeff93486 (127.0.98.2:40171): Not found: Tablet not found: db9d461f21454562ac5f5d627b91a45e
W20260926 01:12:35.542276 15278 leader_election.cc:341] T db9d461f21454562ac5f5d627b91a45e P b193498b4e1a4ca091a9e021e7fa1830 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5e3d9f9c88ff4617b5314f95cb3b5d47 (127.0.98.3:45841): Not found: Tablet not found: db9d461f21454562ac5f5d627b91a45e
I20260926 01:12:35.542289 15278 leader_election.cc:302] T db9d461f21454562ac5f5d627b91a45e P b193498b4e1a4ca091a9e021e7fa1830 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b193498b4e1a4ca091a9e021e7fa1830; no voters: 5e3d9f9c88ff4617b5314f95cb3b5d47, cb9887afb40444c2b400da5aeff93486
I20260926 01:12:35.542353 15670 raft_consensus.cc:2726] T db9d461f21454562ac5f5d627b91a45e P b193498b4e1a4ca091a9e021e7fa1830 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 01:12:35.626686 15671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:35.626469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32949 (local address 127.0.98.1:34765)
0926 01:12:35.626600 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0926 01:12:35.626639 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":81,"threads_started":1}
I20260926 01:12:35.684047 15672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:35.683810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32949 (local address 127.0.98.2:43575)
0926 01:12:35.683946 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0926 01:12:35.683993 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":79,"threads_started":1}
I20260926 01:12:35.814504 15673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:35.814318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32949 (local address 127.0.98.3:56499)
0926 01:12:35.814422 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0926 01:12:35.814459 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":60,"threads_started":1}
I20260926 01:12:36.627331 15674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:36.627134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32949 (local address 127.0.98.1:44665)
0926 01:12:36.627252 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0926 01:12:36.627284 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":63,"threads_started":1}
I20260926 01:12:36.684743 15675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:36.684508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32949 (local address 127.0.98.2:47619)
0926 01:12:36.684656 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 01:12:36.684698 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":70,"threads_started":1}
I20260926 01:12:36.815169 15676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:36.814941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32949 (local address 127.0.98.3:55721)
0926 01:12:36.815083 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0926 01:12:36.815126 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":64,"threads_started":1}
I20260926 01:12:37.628142 15677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:37.627890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32949 (local address 127.0.98.1:35945)
0926 01:12:37.628042 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0926 01:12:37.628090 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":72,"threads_started":1}
I20260926 01:12:37.685438 15678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:37.685228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32949 (local address 127.0.98.2:60305)
0926 01:12:37.685355 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0926 01:12:37.685390 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":67,"threads_started":1}
I20260926 01:12:37.815807 15679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:37.815605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32949 (local address 127.0.98.3:57519)
0926 01:12:37.815724 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0926 01:12:37.815762 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":69,"threads_started":1}
I20260926 01:12:38.253343 15680 raft_consensus.cc:482] T db9d461f21454562ac5f5d627b91a45e P b193498b4e1a4ca091a9e021e7fa1830 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:12:38.253453 15680 raft_consensus.cc:504] T db9d461f21454562ac5f5d627b91a45e P b193498b4e1a4ca091a9e021e7fa1830 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e3d9f9c88ff4617b5314f95cb3b5d47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45841 } } peers { permanent_uuid: "b193498b4e1a4ca091a9e021e7fa1830" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40439 } } peers { permanent_uuid: "cb9887afb40444c2b400da5aeff93486" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40171 } }
I20260926 01:12:38.253589 15680 leader_election.cc:288] T db9d461f21454562ac5f5d627b91a45e P b193498b4e1a4ca091a9e021e7fa1830 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5e3d9f9c88ff4617b5314f95cb3b5d47 (127.0.98.3:45841), cb9887afb40444c2b400da5aeff93486 (127.0.98.2:40171)
W20260926 01:12:38.254005 15278 leader_election.cc:341] T db9d461f21454562ac5f5d627b91a45e P b193498b4e1a4ca091a9e021e7fa1830 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5e3d9f9c88ff4617b5314f95cb3b5d47 (127.0.98.3:45841): Not found: Tablet not found: db9d461f21454562ac5f5d627b91a45e
W20260926 01:12:38.254076 15278 leader_election.cc:341] T db9d461f21454562ac5f5d627b91a45e P b193498b4e1a4ca091a9e021e7fa1830 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cb9887afb40444c2b400da5aeff93486 (127.0.98.2:40171): Not found: Tablet not found: db9d461f21454562ac5f5d627b91a45e
I20260926 01:12:38.254091 15278 leader_election.cc:302] T db9d461f21454562ac5f5d627b91a45e P b193498b4e1a4ca091a9e021e7fa1830 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b193498b4e1a4ca091a9e021e7fa1830; no voters: 5e3d9f9c88ff4617b5314f95cb3b5d47, cb9887afb40444c2b400da5aeff93486
I20260926 01:12:38.254179 15680 raft_consensus.cc:2726] T db9d461f21454562ac5f5d627b91a45e P b193498b4e1a4ca091a9e021e7fa1830 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 01:12:38.629037 15681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:38.628736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32949 (local address 127.0.98.1:59569)
0926 01:12:38.628886 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 01:12:38.628976 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":67,"threads_started":1}
I20260926 01:12:38.686196 15682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:38.685957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32949 (local address 127.0.98.2:35011)
0926 01:12:38.686111 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 01:12:38.686150 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":74,"threads_started":1}
I20260926 01:12:38.816594 15683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:38.816314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32949 (local address 127.0.98.3:45167)
0926 01:12:38.816462 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 01:12:38.816511 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":63,"threads_started":1}
I20260926 01:12:39.629740 15684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:39.629518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32949 (local address 127.0.98.1:34059)
0926 01:12:39.629654 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0926 01:12:39.629694 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":62,"threads_started":1}
I20260926 01:12:39.686880 15685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:39.686652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32949 (local address 127.0.98.2:33049)
0926 01:12:39.686787 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0926 01:12:39.686831 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":62,"threads_started":1}
I20260926 01:12:39.817371 15686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:39.817127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32949 (local address 127.0.98.3:38609)
0926 01:12:39.817270 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 01:12:39.817315 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":71,"threads_started":1}
I20260926 01:12:40.374985 15687 raft_consensus.cc:482] T db9d461f21454562ac5f5d627b91a45e P b193498b4e1a4ca091a9e021e7fa1830 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:12:40.375092 15687 raft_consensus.cc:504] T db9d461f21454562ac5f5d627b91a45e P b193498b4e1a4ca091a9e021e7fa1830 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e3d9f9c88ff4617b5314f95cb3b5d47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45841 } } peers { permanent_uuid: "b193498b4e1a4ca091a9e021e7fa1830" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40439 } } peers { permanent_uuid: "cb9887afb40444c2b400da5aeff93486" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40171 } }
I20260926 01:12:40.375228 15687 leader_election.cc:288] T db9d461f21454562ac5f5d627b91a45e P b193498b4e1a4ca091a9e021e7fa1830 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5e3d9f9c88ff4617b5314f95cb3b5d47 (127.0.98.3:45841), cb9887afb40444c2b400da5aeff93486 (127.0.98.2:40171)
W20260926 01:12:40.375643 15278 leader_election.cc:341] T db9d461f21454562ac5f5d627b91a45e P b193498b4e1a4ca091a9e021e7fa1830 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5e3d9f9c88ff4617b5314f95cb3b5d47 (127.0.98.3:45841): Not found: Tablet not found: db9d461f21454562ac5f5d627b91a45e
W20260926 01:12:40.375713 15278 leader_election.cc:341] T db9d461f21454562ac5f5d627b91a45e P b193498b4e1a4ca091a9e021e7fa1830 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cb9887afb40444c2b400da5aeff93486 (127.0.98.2:40171): Not found: Tablet not found: db9d461f21454562ac5f5d627b91a45e
I20260926 01:12:40.375732 15278 leader_election.cc:302] T db9d461f21454562ac5f5d627b91a45e P b193498b4e1a4ca091a9e021e7fa1830 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b193498b4e1a4ca091a9e021e7fa1830; no voters: 5e3d9f9c88ff4617b5314f95cb3b5d47, cb9887afb40444c2b400da5aeff93486
I20260926 01:12:40.375802 15687 raft_consensus.cc:2726] T db9d461f21454562ac5f5d627b91a45e P b193498b4e1a4ca091a9e021e7fa1830 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 01:12:40.630529 15688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:40.630279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32949 (local address 127.0.98.1:59853)
0926 01:12:40.630430 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 01:12:40.630477 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":62,"threads_started":1}
I20260926 01:12:40.687613 15689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:40.687333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32949 (local address 127.0.98.2:44003)
0926 01:12:40.687505 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0926 01:12:40.687541 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":87,"threads_started":1}
I20260926 01:12:40.818264 15690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:40.818023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32949 (local address 127.0.98.3:42059)
0926 01:12:40.818157 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0926 01:12:40.818200 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":65,"threads_started":1}
I20260926 01:12:41.631294 15691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:41.631043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32949 (local address 127.0.98.1:50855)
0926 01:12:41.631195 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0926 01:12:41.631243 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":69,"threads_started":1}
I20260926 01:12:41.688382 15692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:41.688151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32949 (local address 127.0.98.2:58949)
0926 01:12:41.688284 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0926 01:12:41.688329 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":69,"threads_started":1}
I20260926 01:12:41.819052 15693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:41.818777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32949 (local address 127.0.98.3:43183)
0926 01:12:41.818945 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 01:12:41.818992 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":82,"threads_started":1}
I20260926 01:12:42.631960 15694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:42.631742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32949 (local address 127.0.98.1:41919)
0926 01:12:42.631874 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0926 01:12:42.631914 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":63,"threads_started":1}
I20260926 01:12:42.689167 15695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:42.688937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32949 (local address 127.0.98.2:37535)
0926 01:12:42.689065 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0926 01:12:42.689108 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":61,"threads_started":1}
I20260926 01:12:42.819823 15696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:42.819587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32949 (local address 127.0.98.3:33959)
0926 01:12:42.819721 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0926 01:12:42.819773 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":61,"threads_started":1}
I20260926 01:12:43.178560 15697 raft_consensus.cc:482] T db9d461f21454562ac5f5d627b91a45e P b193498b4e1a4ca091a9e021e7fa1830 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:12:43.178666 15697 raft_consensus.cc:504] T db9d461f21454562ac5f5d627b91a45e P b193498b4e1a4ca091a9e021e7fa1830 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e3d9f9c88ff4617b5314f95cb3b5d47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45841 } } peers { permanent_uuid: "b193498b4e1a4ca091a9e021e7fa1830" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40439 } } peers { permanent_uuid: "cb9887afb40444c2b400da5aeff93486" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40171 } }
I20260926 01:12:43.178822 15697 leader_election.cc:288] T db9d461f21454562ac5f5d627b91a45e P b193498b4e1a4ca091a9e021e7fa1830 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5e3d9f9c88ff4617b5314f95cb3b5d47 (127.0.98.3:45841), cb9887afb40444c2b400da5aeff93486 (127.0.98.2:40171)
W20260926 01:12:43.179343 15278 leader_election.cc:341] T db9d461f21454562ac5f5d627b91a45e P b193498b4e1a4ca091a9e021e7fa1830 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cb9887afb40444c2b400da5aeff93486 (127.0.98.2:40171): Not found: Tablet not found: db9d461f21454562ac5f5d627b91a45e
W20260926 01:12:43.179415 15278 leader_election.cc:341] T db9d461f21454562ac5f5d627b91a45e P b193498b4e1a4ca091a9e021e7fa1830 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5e3d9f9c88ff4617b5314f95cb3b5d47 (127.0.98.3:45841): Not found: Tablet not found: db9d461f21454562ac5f5d627b91a45e
I20260926 01:12:43.179442 15278 leader_election.cc:302] T db9d461f21454562ac5f5d627b91a45e P b193498b4e1a4ca091a9e021e7fa1830 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b193498b4e1a4ca091a9e021e7fa1830; no voters: 5e3d9f9c88ff4617b5314f95cb3b5d47, cb9887afb40444c2b400da5aeff93486
I20260926 01:12:43.179551 15697 raft_consensus.cc:2726] T db9d461f21454562ac5f5d627b91a45e P b193498b4e1a4ca091a9e021e7fa1830 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 01:12:43.632840 15698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:43.632521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32949 (local address 127.0.98.1:39183)
0926 01:12:43.632736 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0926 01:12:43.632782 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":73,"threads_started":1}
I20260926 01:12:43.689981 15699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:43.689702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32949 (local address 127.0.98.2:40129)
0926 01:12:43.689888 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0926 01:12:43.689932 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":62,"threads_started":1}
I20260926 01:12:43.820726 15700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:43.820440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32949 (local address 127.0.98.3:42753)
0926 01:12:43.820604 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 01:12:43.820653 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":93,"threads_started":1}
I20260926 01:12:44.633569 15701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:44.633338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32949 (local address 127.0.98.1:34465)
0926 01:12:44.633486 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 01:12:44.633522 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":62,"threads_started":1}
I20260926 01:12:44.690673 15702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:44.690427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32949 (local address 127.0.98.2:44813)
0926 01:12:44.690581 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 01:12:44.690625 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":76,"threads_started":1}
I20260926 01:12:44.821555 15703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:44.821301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32949 (local address 127.0.98.3:49049)
0926 01:12:44.821457 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 01:12:44.821503 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":82,"threads_started":1}
I20260926 01:12:45.634347 15704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:45.634118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32949 (local address 127.0.98.1:57787)
0926 01:12:45.634260 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0926 01:12:45.634298 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":78,"threads_started":1}
I20260926 01:12:45.690828 15705 raft_consensus.cc:482] T db9d461f21454562ac5f5d627b91a45e P b193498b4e1a4ca091a9e021e7fa1830 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:12:45.690943 15705 raft_consensus.cc:504] T db9d461f21454562ac5f5d627b91a45e P b193498b4e1a4ca091a9e021e7fa1830 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e3d9f9c88ff4617b5314f95cb3b5d47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45841 } } peers { permanent_uuid: "b193498b4e1a4ca091a9e021e7fa1830" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40439 } } peers { permanent_uuid: "cb9887afb40444c2b400da5aeff93486" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40171 } }
I20260926 01:12:45.691147 15705 leader_election.cc:288] T db9d461f21454562ac5f5d627b91a45e P b193498b4e1a4ca091a9e021e7fa1830 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5e3d9f9c88ff4617b5314f95cb3b5d47 (127.0.98.3:45841), cb9887afb40444c2b400da5aeff93486 (127.0.98.2:40171)
I20260926 01:12:45.691489 15706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:45.691257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32949 (local address 127.0.98.2:41915)
0926 01:12:45.691393 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0926 01:12:45.691442 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":54,"threads_started":1}
W20260926 01:12:45.691531 15278 leader_election.cc:341] T db9d461f21454562ac5f5d627b91a45e P b193498b4e1a4ca091a9e021e7fa1830 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cb9887afb40444c2b400da5aeff93486 (127.0.98.2:40171): Not found: Tablet not found: db9d461f21454562ac5f5d627b91a45e
W20260926 01:12:45.691591 15278 leader_election.cc:341] T db9d461f21454562ac5f5d627b91a45e P b193498b4e1a4ca091a9e021e7fa1830 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5e3d9f9c88ff4617b5314f95cb3b5d47 (127.0.98.3:45841): Not found: Tablet not found: db9d461f21454562ac5f5d627b91a45e
I20260926 01:12:45.691620 15278 leader_election.cc:302] T db9d461f21454562ac5f5d627b91a45e P b193498b4e1a4ca091a9e021e7fa1830 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b193498b4e1a4ca091a9e021e7fa1830; no voters: 5e3d9f9c88ff4617b5314f95cb3b5d47, cb9887afb40444c2b400da5aeff93486
I20260926 01:12:45.691701 15705 raft_consensus.cc:2726] T db9d461f21454562ac5f5d627b91a45e P b193498b4e1a4ca091a9e021e7fa1830 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 01:12:45.822378 15707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:45.822123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32949 (local address 127.0.98.3:53343)
0926 01:12:45.822283 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 01:12:45.822329 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":81,"threads_started":1}
I20260926 01:12:46.635176 15708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:46.634894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32949 (local address 127.0.98.1:51581)
0926 01:12:46.635073 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0926 01:12:46.635120 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":66,"threads_started":1}
I20260926 01:12:46.692273 15709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:46.691999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32949 (local address 127.0.98.2:37471)
0926 01:12:46.692153 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 01:12:46.692207 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":67,"threads_started":1}
I20260926 01:12:46.823127 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:46.822926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32949 (local address 127.0.98.3:42643)
0926 01:12:46.823045 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0926 01:12:46.823076 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":63,"threads_started":1}
I20260926 01:12:47.636080 15711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:47.635814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32949 (local address 127.0.98.1:39521)
0926 01:12:47.635986 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0926 01:12:47.636031 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":80,"threads_started":1}
I20260926 01:12:47.693168 15712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:47.692921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32949 (local address 127.0.98.2:45133)
0926 01:12:47.693066 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 01:12:47.693115 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":65,"threads_started":1}
I20260926 01:12:47.822537 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 15231
I20260926 01:12:47.824074 15713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:47.823874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32949 (local address 127.0.98.3:51273)
0926 01:12:47.823992 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0926 01:12:47.824030 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":56,"threads_started":1}
I20260926 01:12:47.827001 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 15394
I20260926 01:12:47.831234 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 15527
2026-09-26T01:12:47Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18132 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 01:12:47 dist-test-slave-7b22 krb5kdc[15716](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 01:12:47 dist-test-slave-7b22 krb5kdc[15716](info): set up 2 sockets
Sep 26 01:12:47 dist-test-slave-7b22 krb5kdc[15716](info): commencing operation
krb5kdc: starting...
W20260926 01:12:48.809723 392 mini_kdc.cc:121] Time spent starting KDC: real 0.959s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/krb5kdc/test-user.keytab.
Sep 26 01:12:48 dist-test-slave-7b22 krb5kdc[15716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385168, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T01:12:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:12:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 01:12:48.932787 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45309
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45309
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:12:49.015942 15729 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:12:49.016242 15729 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:12:49.016269 15729 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:12:49.017624 15729 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 01:12:49.017683 15729 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 01:12:49.017696 15729 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 01:12:49.017710 15729 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:12:49.017722 15729 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:12:49.017735 15729 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:12:49.017746 15729 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:12:49.017756 15729 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:12:49.019474 15729 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45309
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45309
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15729
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:12:49.019742 15729 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:12:49.019982 15729 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 01:12:49.025315 15729 server_base.cc:1034] running on GCE node
W20260926 01:12:49.025332 15735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:12:49.025434 15737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:12:49.025416 15734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:12:49.025980 15729 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:12:49.026297 15729 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:12:49.027437 15729 hybrid_clock.cc:648] HybridClock initialized: now 1790385169027418 us; error 35 us; skew 500 ppm
Sep 26 01:12:49 dist-test-slave-7b22 krb5kdc[15716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385169, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:12:49.030150 15729 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 01:12:49.030575 15729 webserver.cc:460] Webserver started at http://127.0.98.62:42883/ using document root <none> and password file <none>
I20260926 01:12:49.030788 15729 fs_manager.cc:359] Metadata directory not provided
I20260926 01:12:49.030827 15729 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:12:49.030922 15729 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:12:49.031976 15729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/master-0/data/instance:
uuid: "23f32ab05d694df7a44d5716098d6ba7"
format_stamp: "Formatted at 2026-09-26 01:12:49 on dist-test-slave-7b22"
server_key: "ac8ca28dfa705c2c180098f738f866d9"
server_key_iv: "3deeff1277e5b154c414d19b3d6516b1"
server_key_version: "encryptionkey@0"
I20260926 01:12:49.032341 15729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/master-0/wal/instance:
uuid: "23f32ab05d694df7a44d5716098d6ba7"
format_stamp: "Formatted at 2026-09-26 01:12:49 on dist-test-slave-7b22"
server_key: "ac8ca28dfa705c2c180098f738f866d9"
server_key_iv: "3deeff1277e5b154c414d19b3d6516b1"
server_key_version: "encryptionkey@0"
I20260926 01:12:49.034178 15729 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.001s
I20260926 01:12:49.034946 15744 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:12:49.035120 15729 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:12:49.035211 15729 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/master-0/wal
uuid: "23f32ab05d694df7a44d5716098d6ba7"
format_stamp: "Formatted at 2026-09-26 01:12:49 on dist-test-slave-7b22"
server_key: "ac8ca28dfa705c2c180098f738f866d9"
server_key_iv: "3deeff1277e5b154c414d19b3d6516b1"
server_key_version: "encryptionkey@0"
I20260926 01:12:49.035279 15729 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:12:49.051087 15729 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:12:49.053243 15729 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:12:49.053360 15729 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:12:49.057348 15729 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45309
I20260926 01:12:49.057739 15796 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45309 every 8 connection(s)
I20260926 01:12:49.057742 15729 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
I20260926 01:12:49.057936 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 15729
I20260926 01:12:49.058033 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/master-0/wal/instance
I20260926 01:12:49.058243 392 external_mini_cluster.cc:1428] Setting key 86a688a7d05a7606322ab2dd12d24cf3
I20260926 01:12:49.058561 15797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:12:49.062116 15797 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23f32ab05d694df7a44d5716098d6ba7: Bootstrap starting.
I20260926 01:12:49.062827 15797 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 23f32ab05d694df7a44d5716098d6ba7: Neither blocks nor log segments found. Creating new log.
I20260926 01:12:49.063122 15797 log.cc:824] T 00000000000000000000000000000000 P 23f32ab05d694df7a44d5716098d6ba7: Log is configured to *not* fsync() on all Append() calls
Sep 26 01:12:49 dist-test-slave-7b22 krb5kdc[15716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385168, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:12:49.069959 15797 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23f32ab05d694df7a44d5716098d6ba7: No bootstrap required, opened a new log
I20260926 01:12:49.071125 15800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:49.058757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48610 (local address 127.0.98.62:45309)
0926 01:12:49.058934 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 01:12:49.058941 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 01:12:49.058967 (+ 26us) server_negotiation.cc:407] Connection header received
0926 01:12:49.060167 (+ 1200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:12:49.060183 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:12:49.060283 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:12:49.060484 (+ 201us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:12:49.061348 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:12:49.062375 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:12:49.063445 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:12:49.064051 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:12:49.064171 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:12:49.066991 (+ 2820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:12:49.066997 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:12:49.066998 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:12:49.067011 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:12:49.068959 (+ 1948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:12:49.069369 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:12:49.069372 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:12:49.069373 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:12:49.069514 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:12:49.069759 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:12:49.069760 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:12:49.069760 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:12:49.070158 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:12:49.070323 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0926 01:12:49.070820 (+ 497us) server_negotiation.cc:299] Negotiation successful
0926 01:12:49.070953 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":69,"threads_started":1}
I20260926 01:12:49.072381 15797 raft_consensus.cc:348] T 00000000000000000000000000000000 P 23f32ab05d694df7a44d5716098d6ba7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23f32ab05d694df7a44d5716098d6ba7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45309 } }
I20260926 01:12:49.072525 15797 raft_consensus.cc:374] T 00000000000000000000000000000000 P 23f32ab05d694df7a44d5716098d6ba7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:12:49.072543 15797 raft_consensus.cc:729] T 00000000000000000000000000000000 P 23f32ab05d694df7a44d5716098d6ba7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23f32ab05d694df7a44d5716098d6ba7, State: Initialized, Role: FOLLOWER
I20260926 01:12:49.072641 15797 consensus_queue.cc:260] T 00000000000000000000000000000000 P 23f32ab05d694df7a44d5716098d6ba7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23f32ab05d694df7a44d5716098d6ba7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45309 } }
I20260926 01:12:49.072687 15797 raft_consensus.cc:388] T 00000000000000000000000000000000 P 23f32ab05d694df7a44d5716098d6ba7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:12:49.072705 15797 raft_consensus.cc:482] T 00000000000000000000000000000000 P 23f32ab05d694df7a44d5716098d6ba7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:12:49.072731 15797 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 23f32ab05d694df7a44d5716098d6ba7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:12:49.073321 15797 raft_consensus.cc:504] T 00000000000000000000000000000000 P 23f32ab05d694df7a44d5716098d6ba7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23f32ab05d694df7a44d5716098d6ba7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45309 } }
I20260926 01:12:49.073400 15797 leader_election.cc:302] T 00000000000000000000000000000000 P 23f32ab05d694df7a44d5716098d6ba7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 23f32ab05d694df7a44d5716098d6ba7; no voters:
I20260926 01:12:49.073537 15797 leader_election.cc:288] T 00000000000000000000000000000000 P 23f32ab05d694df7a44d5716098d6ba7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:12:49.073624 15802 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 23f32ab05d694df7a44d5716098d6ba7 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:12:49.073760 15797 sys_catalog.cc:564] T 00000000000000000000000000000000 P 23f32ab05d694df7a44d5716098d6ba7 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:12:49.073782 15802 raft_consensus.cc:686] T 00000000000000000000000000000000 P 23f32ab05d694df7a44d5716098d6ba7 [term 1 LEADER]: Becoming Leader. State: Replica: 23f32ab05d694df7a44d5716098d6ba7, State: Running, Role: LEADER
I20260926 01:12:49.073891 15802 consensus_queue.cc:237] T 00000000000000000000000000000000 P 23f32ab05d694df7a44d5716098d6ba7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23f32ab05d694df7a44d5716098d6ba7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45309 } }
I20260926 01:12:49.074281 15803 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23f32ab05d694df7a44d5716098d6ba7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "23f32ab05d694df7a44d5716098d6ba7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23f32ab05d694df7a44d5716098d6ba7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45309 } } }
I20260926 01:12:49.074297 15804 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23f32ab05d694df7a44d5716098d6ba7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 23f32ab05d694df7a44d5716098d6ba7. Latest consensus state: current_term: 1 leader_uuid: "23f32ab05d694df7a44d5716098d6ba7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23f32ab05d694df7a44d5716098d6ba7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45309 } } }
I20260926 01:12:49.074381 15803 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23f32ab05d694df7a44d5716098d6ba7 [sys.catalog]: This master's current role is: LEADER
I20260926 01:12:49.074386 15804 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23f32ab05d694df7a44d5716098d6ba7 [sys.catalog]: This master's current role is: LEADER
W20260926 01:12:49.075589 15818 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 23f32ab05d694df7a44d5716098d6ba7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 01:12:49.075642 15818 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 01:12:49.075695 15811 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:12:49.075862 15811 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:12:49.077399 15811 catalog_manager.cc:1347] Generated new cluster ID: 49a858af3ce94570a7800a2585b79192
I20260926 01:12:49.077457 15811 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 01:12:49.089996 15811 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:12:49.090655 15811 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:12:49.098855 15811 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 23f32ab05d694df7a44d5716098d6ba7: Generated new TSK 0
I20260926 01:12:49.099058 15811 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 01:12:49.129184 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45309
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:12:49.231206 15823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:12:49.231438 15823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:12:49.231472 15823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:12:49.232745 15823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:12:49.232797 15823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:12:49.232837 15823 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 01:12:49.234594 15823 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45309
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15823
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:12:49.234859 15823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:12:49.235085 15823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:12:49.240067 15828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:12:49.240056 15831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:12:49.240124 15829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:12:49.240362 15823 server_base.cc:1034] running on GCE node
I20260926 01:12:49.240593 15823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:12:49.240829 15823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:12:49.241997 15823 hybrid_clock.cc:648] HybridClock initialized: now 1790385169241972 us; error 34 us; skew 500 ppm
Sep 26 01:12:49 dist-test-slave-7b22 krb5kdc[15716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385169, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:12:49.244696 15823 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 01:12:49.245172 15823 webserver.cc:460] Webserver started at http://127.0.98.1:43205/ using document root <none> and password file <none>
I20260926 01:12:49.245401 15823 fs_manager.cc:359] Metadata directory not provided
I20260926 01:12:49.245451 15823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:12:49.245581 15823 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:12:49.246716 15823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-0/data/instance:
uuid: "a44170426a7449ada0cd5417ec05f489"
format_stamp: "Formatted at 2026-09-26 01:12:49 on dist-test-slave-7b22"
server_key: "df9ded4ee0a9fbc082989a9db96fa076"
server_key_iv: "79f3b89faba1690855051b5d290bf1c4"
server_key_version: "encryptionkey@0"
I20260926 01:12:49.247040 15823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-0/wal/instance:
uuid: "a44170426a7449ada0cd5417ec05f489"
format_stamp: "Formatted at 2026-09-26 01:12:49 on dist-test-slave-7b22"
server_key: "df9ded4ee0a9fbc082989a9db96fa076"
server_key_iv: "79f3b89faba1690855051b5d290bf1c4"
server_key_version: "encryptionkey@0"
I20260926 01:12:49.248400 15823 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 01:12:49.249200 15838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:12:49.249379 15823 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 01:12:49.249465 15823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-0/wal
uuid: "a44170426a7449ada0cd5417ec05f489"
format_stamp: "Formatted at 2026-09-26 01:12:49 on dist-test-slave-7b22"
server_key: "df9ded4ee0a9fbc082989a9db96fa076"
server_key_iv: "79f3b89faba1690855051b5d290bf1c4"
server_key_version: "encryptionkey@0"
I20260926 01:12:49.249536 15823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:12:49.282730 15823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:12:49.285068 15823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:12:49.285233 15823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:12:49.285476 15823 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:12:49.285801 15823 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:12:49.285861 15823 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:12:49.285892 15823 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:12:49.285909 15823 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:12:49.291666 15823 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41097
I20260926 01:12:49.291728 15951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41097 every 8 connection(s)
I20260926 01:12:49.292045 15823 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-0/data/info.pb
I20260926 01:12:49.295472 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 15823
I20260926 01:12:49.295585 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-0/wal/instance
I20260926 01:12:49.295770 392 external_mini_cluster.cc:1428] Setting key f5b7c764ca83d1eaa8b2b0b793458a5c
Sep 26 01:12:49 dist-test-slave-7b22 krb5kdc[15716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385169, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:12:49.303321 15800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:49.292950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40553 (local address 127.0.98.62:45309)
0926 01:12:49.293080 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0926 01:12:49.293082 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:12:49.293272 (+ 190us) server_negotiation.cc:407] Connection header received
0926 01:12:49.293794 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:12:49.293795 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:12:49.293807 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:12:49.293880 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:12:49.294876 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:12:49.295439 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:12:49.296774 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:12:49.297050 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:12:49.297125 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:12:49.300095 (+ 2970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:12:49.300097 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:12:49.300097 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:12:49.300105 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:12:49.301778 (+ 1673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:12:49.302321 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:12:49.302323 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:12:49.302323 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:12:49.302383 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:12:49.302625 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:12:49.302626 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:12:49.302626 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:12:49.302749 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:12:49.302805 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 01:12:49.303106 (+ 301us) server_negotiation.cc:299] Negotiation successful
0926 01:12:49.303223 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260926 01:12:49.303510 15954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:49.292886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45309 (local address 127.0.98.1:40553)
0926 01:12:49.293144 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0926 01:12:49.293183 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 01:12:49.293695 (+ 512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:12:49.293962 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:12:49.293966 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:12:49.294564 (+ 598us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:12:49.294776 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:12:49.294782 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:12:49.295604 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:12:49.295606 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:12:49.296678 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:12:49.296681 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:12:49.297140 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:12:49.297141 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:12:49.297287 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:12:49.297904 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:12:49.297915 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:12:49.299975 (+ 2060us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:12:49.301903 (+ 1928us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:12:49.301914 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:12:49.301917 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:12:49.302243 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:12:49.302450 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:12:49.302451 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:12:49.302452 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:12:49.302565 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:12:49.302805 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:12:49.302813 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:12:49.302973 (+ 160us) client_negotiation.cc:769] Sending connection context
0926 01:12:49.303075 (+ 102us) client_negotiation.cc:240] Negotiation successful
0926 01:12:49.303260 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":88,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260926 01:12:49.304111 15952 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45309
I20260926 01:12:49.304306 15952 heartbeater.cc:464] Registering TS with master...
I20260926 01:12:49.304590 15952 heartbeater.cc:511] Master 127.0.98.62:45309 requested a full tablet report, sending...
I20260926 01:12:49.305543 15761 ts_manager.cc:194] Registered new tserver with Master: a44170426a7449ada0cd5417ec05f489 (127.0.98.1:41097)
I20260926 01:12:49.306321 15761 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:40553
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 01:12:49.324643 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45309
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:12:49.408236 15957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:12:49.408455 15957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:12:49.408479 15957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:12:49.409750 15957 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:12:49.409803 15957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:12:49.409857 15957 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 01:12:49.411388 15957 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45309
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15957
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:12:49.411608 15957 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:12:49.411796 15957 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:12:49.416776 15962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:12:49.416833 15957 server_base.cc:1034] running on GCE node
W20260926 01:12:49.416796 15963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:12:49.416841 15965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:12:49.417302 15957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:12:49.417523 15957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:12:49.418720 15957 hybrid_clock.cc:648] HybridClock initialized: now 1790385169418655 us; error 84 us; skew 500 ppm
Sep 26 01:12:49 dist-test-slave-7b22 krb5kdc[15716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385169, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:12:49.421249 15957 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 01:12:49.421748 15957 webserver.cc:460] Webserver started at http://127.0.98.2:39883/ using document root <none> and password file <none>
I20260926 01:12:49.421990 15957 fs_manager.cc:359] Metadata directory not provided
I20260926 01:12:49.422042 15957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:12:49.422169 15957 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:12:49.423216 15957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-1/data/instance:
uuid: "93e20e9f1d2e4ee28d5d0891a70aa96e"
format_stamp: "Formatted at 2026-09-26 01:12:49 on dist-test-slave-7b22"
server_key: "792d916d3335ac55de814f8a8c732b05"
server_key_iv: "a3b5ba14f34ab0797dc72053300ca2e5"
server_key_version: "encryptionkey@0"
I20260926 01:12:49.423540 15957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-1/wal/instance:
uuid: "93e20e9f1d2e4ee28d5d0891a70aa96e"
format_stamp: "Formatted at 2026-09-26 01:12:49 on dist-test-slave-7b22"
server_key: "792d916d3335ac55de814f8a8c732b05"
server_key_iv: "a3b5ba14f34ab0797dc72053300ca2e5"
server_key_version: "encryptionkey@0"
I20260926 01:12:49.424856 15957 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 01:12:49.425698 15972 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:12:49.425909 15957 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:12:49.426008 15957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-1/wal
uuid: "93e20e9f1d2e4ee28d5d0891a70aa96e"
format_stamp: "Formatted at 2026-09-26 01:12:49 on dist-test-slave-7b22"
server_key: "792d916d3335ac55de814f8a8c732b05"
server_key_iv: "a3b5ba14f34ab0797dc72053300ca2e5"
server_key_version: "encryptionkey@0"
I20260926 01:12:49.426074 15957 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:12:49.445114 15957 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:12:49.447350 15957 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:12:49.447494 15957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:12:49.447727 15957 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:12:49.448024 15957 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:12:49.448057 15957 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:12:49.448086 15957 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:12:49.448103 15957 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:12:49.454087 15957 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39747
I20260926 01:12:49.454136 16085 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39747 every 8 connection(s)
I20260926 01:12:49.454460 15957 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-1/data/info.pb
I20260926 01:12:49.460194 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 15957
I20260926 01:12:49.460315 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-1/wal/instance
I20260926 01:12:49.460544 392 external_mini_cluster.cc:1428] Setting key 5307bb47191f867ff4ab65a0a659012f
Sep 26 01:12:49 dist-test-slave-7b22 krb5kdc[15716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385169, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:12:49.464128 15800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:49.455159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53967 (local address 127.0.98.62:45309)
0926 01:12:49.455216 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 01:12:49.455218 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:12:49.455386 (+ 168us) server_negotiation.cc:407] Connection header received
0926 01:12:49.455909 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:12:49.455910 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:12:49.455922 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:12:49.455971 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:12:49.456916 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:12:49.457414 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:12:49.458422 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:12:49.458611 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:12:49.458668 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:12:49.461202 (+ 2534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:12:49.461204 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:12:49.461205 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:12:49.461212 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:12:49.462675 (+ 1463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:12:49.463170 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:12:49.463171 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:12:49.463171 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:12:49.463224 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:12:49.463473 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:12:49.463474 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:12:49.463474 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:12:49.463575 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:12:49.463629 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 01:12:49.463937 (+ 308us) server_negotiation.cc:299] Negotiation successful
0926 01:12:49.464032 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 01:12:49.464324 16088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:49.455123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45309 (local address 127.0.98.2:53967)
0926 01:12:49.455261 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0926 01:12:49.455308 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 01:12:49.455812 (+ 504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:12:49.456049 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:12:49.456052 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:12:49.456632 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:12:49.456826 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:12:49.456831 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:12:49.457495 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:12:49.457496 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:12:49.458339 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:12:49.458341 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:12:49.458680 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:12:49.458681 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:12:49.458801 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:12:49.459356 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:12:49.459365 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:12:49.461098 (+ 1733us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:12:49.462767 (+ 1669us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:12:49.462776 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:12:49.462779 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:12:49.463098 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:12:49.463278 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:12:49.463279 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:12:49.463279 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:12:49.463413 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:12:49.463656 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:12:49.463663 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:12:49.463810 (+ 147us) client_negotiation.cc:769] Sending connection context
0926 01:12:49.463906 (+ 96us) client_negotiation.cc:240] Negotiation successful
0926 01:12:49.464079 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":64,"threads_started":1}
I20260926 01:12:49.464816 16086 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45309
I20260926 01:12:49.464915 16086 heartbeater.cc:464] Registering TS with master...
I20260926 01:12:49.465102 16086 heartbeater.cc:511] Master 127.0.98.62:45309 requested a full tablet report, sending...
I20260926 01:12:49.465507 15761 ts_manager.cc:194] Registered new tserver with Master: 93e20e9f1d2e4ee28d5d0891a70aa96e (127.0.98.2:39747)
I20260926 01:12:49.466058 15761 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:53967
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 01:12:49.485642 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45309
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:12:49.571594 16091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:12:49.571822 16091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:12:49.571848 16091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:12:49.573069 16091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:12:49.573122 16091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:12:49.573148 16091 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 01:12:49.574787 16091 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45309
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16091
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:12:49.575006 16091 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:12:49.575233 16091 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:12:49.580054 16099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:12:49.580034 16096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:12:49.580036 16097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:12:49.580457 16091 server_base.cc:1034] running on GCE node
I20260926 01:12:49.580647 16091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:12:49.580881 16091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:12:49.582026 16091 hybrid_clock.cc:648] HybridClock initialized: now 1790385169582012 us; error 34 us; skew 500 ppm
Sep 26 01:12:49 dist-test-slave-7b22 krb5kdc[15716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385169, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:12:49.585167 16091 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 01:12:49.585669 16091 webserver.cc:460] Webserver started at http://127.0.98.3:43609/ using document root <none> and password file <none>
I20260926 01:12:49.585944 16091 fs_manager.cc:359] Metadata directory not provided
I20260926 01:12:49.585995 16091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:12:49.586107 16091 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:12:49.587189 16091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-2/data/instance:
uuid: "32a292318dbc49b8a386537088d5931e"
format_stamp: "Formatted at 2026-09-26 01:12:49 on dist-test-slave-7b22"
server_key: "40190f19815e04d52f09f97fef649f8e"
server_key_iv: "f9942246e0fa6846e7aeeb7ee71e1f3e"
server_key_version: "encryptionkey@0"
I20260926 01:12:49.587522 16091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-2/wal/instance:
uuid: "32a292318dbc49b8a386537088d5931e"
format_stamp: "Formatted at 2026-09-26 01:12:49 on dist-test-slave-7b22"
server_key: "40190f19815e04d52f09f97fef649f8e"
server_key_iv: "f9942246e0fa6846e7aeeb7ee71e1f3e"
server_key_version: "encryptionkey@0"
I20260926 01:12:49.588843 16091 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 01:12:49.589635 16106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:12:49.589843 16091 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 01:12:49.589936 16091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-2/wal
uuid: "32a292318dbc49b8a386537088d5931e"
format_stamp: "Formatted at 2026-09-26 01:12:49 on dist-test-slave-7b22"
server_key: "40190f19815e04d52f09f97fef649f8e"
server_key_iv: "f9942246e0fa6846e7aeeb7ee71e1f3e"
server_key_version: "encryptionkey@0"
I20260926 01:12:49.590001 16091 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:12:49.627840 16091 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:12:49.630997 16091 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:12:49.631183 16091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:12:49.631450 16091 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:12:49.631765 16091 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:12:49.631799 16091 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:12:49.631829 16091 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:12:49.631853 16091 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:12:49.637933 16091 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42549
I20260926 01:12:49.637987 16219 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42549 every 8 connection(s)
I20260926 01:12:49.638324 16091 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-2/data/info.pb
I20260926 01:12:49.641317 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 16091
I20260926 01:12:49.641425 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-2/wal/instance
I20260926 01:12:49.641597 392 external_mini_cluster.cc:1428] Setting key 6a332533ab742eff0523d355c54eb5a4
Sep 26 01:12:49 dist-test-slave-7b22 krb5kdc[15716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385169, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:12:49.648728 15800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:49.639116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49433 (local address 127.0.98.62:45309)
0926 01:12:49.639176 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 01:12:49.639178 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:12:49.639370 (+ 192us) server_negotiation.cc:407] Connection header received
0926 01:12:49.639881 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:12:49.639882 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:12:49.639894 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:12:49.639946 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:12:49.640942 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:12:49.641605 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:12:49.643352 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:12:49.643562 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:12:49.643626 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:12:49.646084 (+ 2458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:12:49.646085 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:12:49.646086 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:12:49.646094 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:12:49.647404 (+ 1310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:12:49.647825 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:12:49.647826 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:12:49.647826 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:12:49.647890 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:12:49.648125 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:12:49.648126 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:12:49.648126 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:12:49.648253 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:12:49.648316 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 01:12:49.648574 (+ 258us) server_negotiation.cc:299] Negotiation successful
0926 01:12:49.648652 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260926 01:12:49.648877 16222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:49.639043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45309 (local address 127.0.98.3:49433)
0926 01:12:49.639235 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0926 01:12:49.639277 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 01:12:49.639787 (+ 510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:12:49.640017 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:12:49.640021 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:12:49.640630 (+ 609us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:12:49.640840 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:12:49.640846 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:12:49.641926 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:12:49.641928 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:12:49.643243 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:12:49.643249 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:12:49.643663 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:12:49.643664 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:12:49.643785 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:12:49.644363 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:12:49.644372 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:12:49.645906 (+ 1534us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:12:49.647484 (+ 1578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:12:49.647492 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:12:49.647494 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:12:49.647742 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:12:49.647962 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:12:49.647963 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:12:49.647963 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:12:49.648074 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:12:49.648314 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:12:49.648318 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:12:49.648452 (+ 134us) client_negotiation.cc:769] Sending connection context
0926 01:12:49.648535 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 01:12:49.648671 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":78,"threads_started":1}
I20260926 01:12:49.649317 16220 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45309
I20260926 01:12:49.649410 16220 heartbeater.cc:464] Registering TS with master...
I20260926 01:12:49.649595 16220 heartbeater.cc:511] Master 127.0.98.62:45309 requested a full tablet report, sending...
I20260926 01:12:49.650071 15761 ts_manager.cc:194] Registered new tserver with Master: 32a292318dbc49b8a386537088d5931e (127.0.98.3:42549)
I20260926 01:12:49.650473 15761 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:49433
I20260926 01:12:49.654083 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 01:12:49.661898 15800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:49.655513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48624 (local address 127.0.98.62:45309)
0926 01:12:49.655577 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 01:12:49.655578 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 01:12:49.655589 (+ 11us) server_negotiation.cc:407] Connection header received
0926 01:12:49.655617 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:12:49.655617 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:12:49.655628 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:12:49.655710 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:12:49.656304 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:12:49.656834 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:12:49.657642 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:12:49.657884 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:12:49.657950 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:12:49.659502 (+ 1552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:12:49.659504 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:12:49.659504 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:12:49.659511 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:12:49.660804 (+ 1293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:12:49.661159 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:12:49.661159 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:12:49.661159 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:12:49.661236 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:12:49.661382 (+ 146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:12:49.661382 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:12:49.661382 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:12:49.661528 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:12:49.661601 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 01:12:49.661715 (+ 114us) server_negotiation.cc:299] Negotiation successful
0926 01:12:49.661775 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 01:12:49.663650 15761 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48624:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:12:49.665264 15761 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:12:49.671494 16231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:49.667467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41097 (local address 127.0.0.1:40486)
0926 01:12:49.667631 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 01:12:49.667669 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 01:12:49.667732 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:12:49.668013 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:12:49.668015 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:12:49.668017 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:12:49.668108 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:12:49.668110 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:12:49.669125 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:12:49.669126 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:12:49.670379 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:12:49.670381 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:12:49.671149 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:12:49.671150 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:12:49.671243 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:12:49.671259 (+ 16us) client_negotiation.cc:769] Sending connection context
0926 01:12:49.671345 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 01:12:49.671404 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":98,"threads_started":1}
I20260926 01:12:49.671504 16235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:49.667669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40486 (local address 127.0.98.1:41097)
0926 01:12:49.667835 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0926 01:12:49.667839 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 01:12:49.667849 (+ 10us) server_negotiation.cc:407] Connection header received
0926 01:12:49.667888 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:12:49.667891 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:12:49.667906 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:12:49.668008 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:12:49.668188 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:12:49.669031 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:12:49.670471 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:12:49.671074 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:12:49.671141 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:12:49.671203 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 01:12:49.671357 (+ 154us) server_negotiation.cc:299] Negotiation successful
0926 01:12:49.671400 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":70,"threads_started":1}
I20260926 01:12:49.671844 16234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:49.667467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39747 (local address 127.0.0.1:59500)
0926 01:12:49.667832 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0926 01:12:49.667840 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 01:12:49.667884 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:12:49.668114 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:12:49.668115 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:12:49.668118 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:12:49.668174 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:12:49.668176 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:12:49.669533 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:12:49.669534 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:12:49.670622 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:12:49.670624 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:12:49.671591 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:12:49.671592 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:12:49.671660 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:12:49.671665 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 01:12:49.671725 (+ 60us) client_negotiation.cc:240] Negotiation successful
0926 01:12:49.671772 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"mutex_wait_us":2,"thread_start_us":59,"threads_started":1}
I20260926 01:12:49.671885 16233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:49.667576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59500 (local address 127.0.98.2:39747)
0926 01:12:49.667776 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 01:12:49.667781 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 01:12:49.667942 (+ 161us) server_negotiation.cc:407] Connection header received
0926 01:12:49.667989 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:12:49.667991 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:12:49.668036 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:12:49.668114 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:12:49.668301 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:12:49.669416 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:12:49.670703 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:12:49.671415 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:12:49.671496 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:12:49.671552 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 01:12:49.671754 (+ 202us) server_negotiation.cc:299] Negotiation successful
0926 01:12:49.671801 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":90,"threads_started":1}
I20260926 01:12:49.672287 16232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:49.667663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42549 (local address 127.0.0.1:36558)
0926 01:12:49.667860 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0926 01:12:49.667868 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 01:12:49.667906 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:12:49.668260 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:12:49.668261 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:12:49.668264 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:12:49.668301 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:12:49.668302 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:12:49.669920 (+ 1618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:12:49.669921 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:12:49.670910 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:12:49.670913 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:12:49.672109 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:12:49.672110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:12:49.672155 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:12:49.672159 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 01:12:49.672207 (+ 48us) client_negotiation.cc:240] Negotiation successful
0926 01:12:49.672238 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":66,"threads_started":1}
I20260926 01:12:49.672338 16236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:49.667860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36558 (local address 127.0.98.3:42549)
0926 01:12:49.668034 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 01:12:49.668038 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 01:12:49.668046 (+ 8us) server_negotiation.cc:407] Connection header received
0926 01:12:49.668081 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:12:49.668083 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:12:49.668097 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:12:49.668172 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:12:49.668372 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:12:49.669428 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:12:49.671003 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:12:49.671877 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:12:49.672105 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:12:49.672166 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 01:12:49.672218 (+ 52us) server_negotiation.cc:299] Negotiation successful
0926 01:12:49.672262 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":81,"threads_started":1}
I20260926 01:12:49.672544 15886 tablet_service.cc:1467] Processing CreateTablet for tablet 6511a23cd8644897861ff82bab054511 (DEFAULT_TABLE table=test-table [id=4798ffee5fbe4b6191fb8e9dac55d5c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:12:49.672926 15886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6511a23cd8644897861ff82bab054511. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:12:49.672920 16154 tablet_service.cc:1467] Processing CreateTablet for tablet 6511a23cd8644897861ff82bab054511 (DEFAULT_TABLE table=test-table [id=4798ffee5fbe4b6191fb8e9dac55d5c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:12:49.673208 16154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6511a23cd8644897861ff82bab054511. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:12:49.676440 16238 tablet_bootstrap.cc:492] T 6511a23cd8644897861ff82bab054511 P a44170426a7449ada0cd5417ec05f489: Bootstrap starting.
I20260926 01:12:49.676687 16237 tablet_bootstrap.cc:492] T 6511a23cd8644897861ff82bab054511 P 32a292318dbc49b8a386537088d5931e: Bootstrap starting.
I20260926 01:12:49.677445 16237 tablet_bootstrap.cc:654] T 6511a23cd8644897861ff82bab054511 P 32a292318dbc49b8a386537088d5931e: Neither blocks nor log segments found. Creating new log.
I20260926 01:12:49.677366 16020 tablet_service.cc:1467] Processing CreateTablet for tablet 6511a23cd8644897861ff82bab054511 (DEFAULT_TABLE table=test-table [id=4798ffee5fbe4b6191fb8e9dac55d5c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:12:49.677729 16020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6511a23cd8644897861ff82bab054511. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:12:49.678047 16237 log.cc:824] T 6511a23cd8644897861ff82bab054511 P 32a292318dbc49b8a386537088d5931e: Log is configured to *not* fsync() on all Append() calls
I20260926 01:12:49.678455 16238 tablet_bootstrap.cc:654] T 6511a23cd8644897861ff82bab054511 P a44170426a7449ada0cd5417ec05f489: Neither blocks nor log segments found. Creating new log.
I20260926 01:12:49.678890 16237 tablet_bootstrap.cc:492] T 6511a23cd8644897861ff82bab054511 P 32a292318dbc49b8a386537088d5931e: No bootstrap required, opened a new log
I20260926 01:12:49.678962 16237 ts_tablet_manager.cc:1397] T 6511a23cd8644897861ff82bab054511 P 32a292318dbc49b8a386537088d5931e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 01:12:49.678989 16238 log.cc:824] T 6511a23cd8644897861ff82bab054511 P a44170426a7449ada0cd5417ec05f489: Log is configured to *not* fsync() on all Append() calls
I20260926 01:12:49.679651 16238 tablet_bootstrap.cc:492] T 6511a23cd8644897861ff82bab054511 P a44170426a7449ada0cd5417ec05f489: No bootstrap required, opened a new log
I20260926 01:12:49.679711 16238 ts_tablet_manager.cc:1397] T 6511a23cd8644897861ff82bab054511 P a44170426a7449ada0cd5417ec05f489: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 01:12:49.680926 16237 raft_consensus.cc:348] T 6511a23cd8644897861ff82bab054511 P 32a292318dbc49b8a386537088d5931e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44170426a7449ada0cd5417ec05f489" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41097 } } peers { permanent_uuid: "93e20e9f1d2e4ee28d5d0891a70aa96e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39747 } } peers { permanent_uuid: "32a292318dbc49b8a386537088d5931e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42549 } }
I20260926 01:12:49.681073 16241 tablet_bootstrap.cc:492] T 6511a23cd8644897861ff82bab054511 P 93e20e9f1d2e4ee28d5d0891a70aa96e: Bootstrap starting.
I20260926 01:12:49.681227 16237 raft_consensus.cc:374] T 6511a23cd8644897861ff82bab054511 P 32a292318dbc49b8a386537088d5931e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:12:49.681149 16238 raft_consensus.cc:348] T 6511a23cd8644897861ff82bab054511 P a44170426a7449ada0cd5417ec05f489 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44170426a7449ada0cd5417ec05f489" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41097 } } peers { permanent_uuid: "93e20e9f1d2e4ee28d5d0891a70aa96e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39747 } } peers { permanent_uuid: "32a292318dbc49b8a386537088d5931e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42549 } }
I20260926 01:12:49.681299 16238 raft_consensus.cc:374] T 6511a23cd8644897861ff82bab054511 P a44170426a7449ada0cd5417ec05f489 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:12:49.681349 16237 raft_consensus.cc:729] T 6511a23cd8644897861ff82bab054511 P 32a292318dbc49b8a386537088d5931e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32a292318dbc49b8a386537088d5931e, State: Initialized, Role: FOLLOWER
I20260926 01:12:49.681494 16238 raft_consensus.cc:729] T 6511a23cd8644897861ff82bab054511 P a44170426a7449ada0cd5417ec05f489 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a44170426a7449ada0cd5417ec05f489, State: Initialized, Role: FOLLOWER
I20260926 01:12:49.681517 16237 consensus_queue.cc:260] T 6511a23cd8644897861ff82bab054511 P 32a292318dbc49b8a386537088d5931e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44170426a7449ada0cd5417ec05f489" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41097 } } peers { permanent_uuid: "93e20e9f1d2e4ee28d5d0891a70aa96e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39747 } } peers { permanent_uuid: "32a292318dbc49b8a386537088d5931e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42549 } }
I20260926 01:12:49.681677 16238 consensus_queue.cc:260] T 6511a23cd8644897861ff82bab054511 P a44170426a7449ada0cd5417ec05f489 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44170426a7449ada0cd5417ec05f489" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41097 } } peers { permanent_uuid: "93e20e9f1d2e4ee28d5d0891a70aa96e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39747 } } peers { permanent_uuid: "32a292318dbc49b8a386537088d5931e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42549 } }
I20260926 01:12:49.681782 16241 tablet_bootstrap.cc:654] T 6511a23cd8644897861ff82bab054511 P 93e20e9f1d2e4ee28d5d0891a70aa96e: Neither blocks nor log segments found. Creating new log.
I20260926 01:12:49.681804 16237 ts_tablet_manager.cc:1428] T 6511a23cd8644897861ff82bab054511 P 32a292318dbc49b8a386537088d5931e: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260926 01:12:49.681962 16220 heartbeater.cc:503] Master 127.0.98.62:45309 was elected leader, sending a full tablet report...
I20260926 01:12:49.682001 16238 ts_tablet_manager.cc:1428] T 6511a23cd8644897861ff82bab054511 P a44170426a7449ada0cd5417ec05f489: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 01:12:49.682111 15952 heartbeater.cc:503] Master 127.0.98.62:45309 was elected leader, sending a full tablet report...
I20260926 01:12:49.682399 16241 log.cc:824] T 6511a23cd8644897861ff82bab054511 P 93e20e9f1d2e4ee28d5d0891a70aa96e: Log is configured to *not* fsync() on all Append() calls
I20260926 01:12:49.683326 16241 tablet_bootstrap.cc:492] T 6511a23cd8644897861ff82bab054511 P 93e20e9f1d2e4ee28d5d0891a70aa96e: No bootstrap required, opened a new log
I20260926 01:12:49.683388 16241 ts_tablet_manager.cc:1397] T 6511a23cd8644897861ff82bab054511 P 93e20e9f1d2e4ee28d5d0891a70aa96e: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 01:12:49.684921 16241 raft_consensus.cc:348] T 6511a23cd8644897861ff82bab054511 P 93e20e9f1d2e4ee28d5d0891a70aa96e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44170426a7449ada0cd5417ec05f489" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41097 } } peers { permanent_uuid: "93e20e9f1d2e4ee28d5d0891a70aa96e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39747 } } peers { permanent_uuid: "32a292318dbc49b8a386537088d5931e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42549 } }
I20260926 01:12:49.685101 16241 raft_consensus.cc:374] T 6511a23cd8644897861ff82bab054511 P 93e20e9f1d2e4ee28d5d0891a70aa96e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:12:49.685178 16241 raft_consensus.cc:729] T 6511a23cd8644897861ff82bab054511 P 93e20e9f1d2e4ee28d5d0891a70aa96e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93e20e9f1d2e4ee28d5d0891a70aa96e, State: Initialized, Role: FOLLOWER
I20260926 01:12:49.685292 16241 consensus_queue.cc:260] T 6511a23cd8644897861ff82bab054511 P 93e20e9f1d2e4ee28d5d0891a70aa96e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44170426a7449ada0cd5417ec05f489" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41097 } } peers { permanent_uuid: "93e20e9f1d2e4ee28d5d0891a70aa96e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39747 } } peers { permanent_uuid: "32a292318dbc49b8a386537088d5931e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42549 } }
I20260926 01:12:49.685535 16241 ts_tablet_manager.cc:1428] T 6511a23cd8644897861ff82bab054511 P 93e20e9f1d2e4ee28d5d0891a70aa96e: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260926 01:12:49.685717 16086 heartbeater.cc:503] Master 127.0.98.62:45309 was elected leader, sending a full tablet report...
W20260926 01:12:49.705024 16087 tablet.cc:2369] T 6511a23cd8644897861ff82bab054511 P 93e20e9f1d2e4ee28d5d0891a70aa96e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 01:12:49.792922 15953 tablet.cc:2369] T 6511a23cd8644897861ff82bab054511 P a44170426a7449ada0cd5417ec05f489: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 01:12:49.888976 16221 tablet.cc:2369] T 6511a23cd8644897861ff82bab054511 P 32a292318dbc49b8a386537088d5931e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 01:12:50.069782 16242 raft_consensus.cc:482] T 6511a23cd8644897861ff82bab054511 P 32a292318dbc49b8a386537088d5931e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:12:50.070127 16242 raft_consensus.cc:504] T 6511a23cd8644897861ff82bab054511 P 32a292318dbc49b8a386537088d5931e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44170426a7449ada0cd5417ec05f489" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41097 } } peers { permanent_uuid: "93e20e9f1d2e4ee28d5d0891a70aa96e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39747 } } peers { permanent_uuid: "32a292318dbc49b8a386537088d5931e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42549 } }
I20260926 01:12:50.070721 16242 leader_election.cc:288] T 6511a23cd8644897861ff82bab054511 P 32a292318dbc49b8a386537088d5931e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a44170426a7449ada0cd5417ec05f489 (127.0.98.1:41097), 93e20e9f1d2e4ee28d5d0891a70aa96e (127.0.98.2:39747)
I20260926 01:12:50.073897 16235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:50.071038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45521 (local address 127.0.98.1:41097)
0926 01:12:50.071130 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0926 01:12:50.071132 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:12:50.071146 (+ 14us) server_negotiation.cc:407] Connection header received
0926 01:12:50.071313 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:12:50.071315 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:12:50.071328 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:12:50.071423 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:12:50.071603 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:12:50.072121 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:12:50.073144 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:12:50.073512 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:12:50.073573 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:12:50.073585 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0926 01:12:50.073741 (+ 156us) server_negotiation.cc:299] Negotiation successful
0926 01:12:50.073786 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 01:12:50.073897 16222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:50.070975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41097 (local address 127.0.98.3:45521)
0926 01:12:50.071065 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0926 01:12:50.071081 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 01:12:50.071143 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:12:50.071459 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:12:50.071461 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:12:50.071464 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:12:50.071536 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:12:50.071538 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:12:50.072197 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:12:50.072199 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:12:50.073047 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:12:50.073049 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:12:50.073585 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:12:50.073587 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:12:50.073666 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:12:50.073669 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 01:12:50.073740 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 01:12:50.073790 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260926 01:12:50.074131 16246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:50.070984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39747 (local address 127.0.98.3:36521)
0926 01:12:50.071331 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0926 01:12:50.071339 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 01:12:50.071389 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:12:50.071551 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:12:50.071552 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:12:50.071555 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:12:50.071647 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:12:50.071649 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:12:50.072295 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:12:50.072296 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:12:50.073181 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:12:50.073184 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:12:50.073797 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:12:50.073798 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:12:50.073914 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:12:50.073920 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 01:12:50.073970 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 01:12:50.074023 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"mutex_wait_us":156,"thread_start_us":125,"threads_started":1}
I20260926 01:12:50.074121 16233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:50.071061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36521 (local address 127.0.98.2:39747)
0926 01:12:50.071126 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 01:12:50.071128 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:12:50.071440 (+ 312us) server_negotiation.cc:407] Connection header received
0926 01:12:50.071478 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:12:50.071479 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:12:50.071493 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:12:50.071541 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:12:50.071703 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:12:50.072220 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:12:50.073279 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:12:50.073680 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:12:50.073762 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:12:50.073776 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0926 01:12:50.073985 (+ 209us) server_negotiation.cc:299] Negotiation successful
0926 01:12:50.074041 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 01:12:50.074306 15906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6511a23cd8644897861ff82bab054511" candidate_uuid: "32a292318dbc49b8a386537088d5931e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a44170426a7449ada0cd5417ec05f489" is_pre_election: true
I20260926 01:12:50.074517 15906 raft_consensus.cc:2454] T 6511a23cd8644897861ff82bab054511 P a44170426a7449ada0cd5417ec05f489 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32a292318dbc49b8a386537088d5931e in term 0.
I20260926 01:12:50.074800 16108 leader_election.cc:302] T 6511a23cd8644897861ff82bab054511 P 32a292318dbc49b8a386537088d5931e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32a292318dbc49b8a386537088d5931e, a44170426a7449ada0cd5417ec05f489; no voters:
I20260926 01:12:50.074823 16040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6511a23cd8644897861ff82bab054511" candidate_uuid: "32a292318dbc49b8a386537088d5931e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93e20e9f1d2e4ee28d5d0891a70aa96e" is_pre_election: true
I20260926 01:12:50.074966 16040 raft_consensus.cc:2454] T 6511a23cd8644897861ff82bab054511 P 93e20e9f1d2e4ee28d5d0891a70aa96e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32a292318dbc49b8a386537088d5931e in term 0.
I20260926 01:12:50.074960 16242 raft_consensus.cc:2781] T 6511a23cd8644897861ff82bab054511 P 32a292318dbc49b8a386537088d5931e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:12:50.075011 16242 raft_consensus.cc:482] T 6511a23cd8644897861ff82bab054511 P 32a292318dbc49b8a386537088d5931e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:12:50.075034 16242 raft_consensus.cc:3037] T 6511a23cd8644897861ff82bab054511 P 32a292318dbc49b8a386537088d5931e [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:12:50.075910 16242 raft_consensus.cc:504] T 6511a23cd8644897861ff82bab054511 P 32a292318dbc49b8a386537088d5931e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44170426a7449ada0cd5417ec05f489" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41097 } } peers { permanent_uuid: "93e20e9f1d2e4ee28d5d0891a70aa96e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39747 } } peers { permanent_uuid: "32a292318dbc49b8a386537088d5931e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42549 } }
I20260926 01:12:50.076054 16242 leader_election.cc:288] T 6511a23cd8644897861ff82bab054511 P 32a292318dbc49b8a386537088d5931e [CANDIDATE]: Term 1 election: Requested vote from peers a44170426a7449ada0cd5417ec05f489 (127.0.98.1:41097), 93e20e9f1d2e4ee28d5d0891a70aa96e (127.0.98.2:39747)
I20260926 01:12:50.076278 16040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6511a23cd8644897861ff82bab054511" candidate_uuid: "32a292318dbc49b8a386537088d5931e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93e20e9f1d2e4ee28d5d0891a70aa96e"
I20260926 01:12:50.076364 16040 raft_consensus.cc:3037] T 6511a23cd8644897861ff82bab054511 P 93e20e9f1d2e4ee28d5d0891a70aa96e [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:12:50.076526 15906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6511a23cd8644897861ff82bab054511" candidate_uuid: "32a292318dbc49b8a386537088d5931e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a44170426a7449ada0cd5417ec05f489"
I20260926 01:12:50.076675 15906 raft_consensus.cc:3037] T 6511a23cd8644897861ff82bab054511 P a44170426a7449ada0cd5417ec05f489 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:12:50.077144 16040 raft_consensus.cc:2454] T 6511a23cd8644897861ff82bab054511 P 93e20e9f1d2e4ee28d5d0891a70aa96e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32a292318dbc49b8a386537088d5931e in term 1.
I20260926 01:12:50.077333 16107 leader_election.cc:302] T 6511a23cd8644897861ff82bab054511 P 32a292318dbc49b8a386537088d5931e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32a292318dbc49b8a386537088d5931e, 93e20e9f1d2e4ee28d5d0891a70aa96e; no voters:
I20260926 01:12:50.077454 16242 raft_consensus.cc:2781] T 6511a23cd8644897861ff82bab054511 P 32a292318dbc49b8a386537088d5931e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:12:50.077628 16242 raft_consensus.cc:686] T 6511a23cd8644897861ff82bab054511 P 32a292318dbc49b8a386537088d5931e [term 1 LEADER]: Becoming Leader. State: Replica: 32a292318dbc49b8a386537088d5931e, State: Running, Role: LEADER
I20260926 01:12:50.077739 16242 consensus_queue.cc:237] T 6511a23cd8644897861ff82bab054511 P 32a292318dbc49b8a386537088d5931e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44170426a7449ada0cd5417ec05f489" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41097 } } peers { permanent_uuid: "93e20e9f1d2e4ee28d5d0891a70aa96e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39747 } } peers { permanent_uuid: "32a292318dbc49b8a386537088d5931e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42549 } }
I20260926 01:12:50.077800 15906 raft_consensus.cc:2454] T 6511a23cd8644897861ff82bab054511 P a44170426a7449ada0cd5417ec05f489 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32a292318dbc49b8a386537088d5931e in term 1.
I20260926 01:12:50.078687 15761 catalog_manager.cc:5495] T 6511a23cd8644897861ff82bab054511 P 32a292318dbc49b8a386537088d5931e reported cstate change: term changed from 0 to 1, leader changed from <none> to 32a292318dbc49b8a386537088d5931e (127.0.98.3). New cstate: current_term: 1 leader_uuid: "32a292318dbc49b8a386537088d5931e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44170426a7449ada0cd5417ec05f489" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93e20e9f1d2e4ee28d5d0891a70aa96e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32a292318dbc49b8a386537088d5931e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42549 } health_report { overall_health: HEALTHY } } }
I20260926 01:12:50.121186 16236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:50.118522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36572 (local address 127.0.98.3:42549)
0926 01:12:50.118648 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0926 01:12:50.118650 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:12:50.118663 (+ 13us) server_negotiation.cc:407] Connection header received
0926 01:12:50.118717 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:12:50.118718 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:12:50.118746 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:12:50.118846 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:12:50.119169 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:12:50.119668 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:12:50.120458 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:12:50.120648 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:12:50.120705 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:12:50.120860 (+ 155us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:12:50.120966 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:12:50.121042 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 01:12:50.121079 (+ 37us) server_negotiation.cc:299] Negotiation successful
0926 01:12:50.121102 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260926 01:12:50.122931 16040 raft_consensus.cc:1261] T 6511a23cd8644897861ff82bab054511 P 93e20e9f1d2e4ee28d5d0891a70aa96e [term 1 FOLLOWER]: Refusing update from remote peer 32a292318dbc49b8a386537088d5931e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:12:50.122932 15906 raft_consensus.cc:1261] T 6511a23cd8644897861ff82bab054511 P a44170426a7449ada0cd5417ec05f489 [term 1 FOLLOWER]: Refusing update from remote peer 32a292318dbc49b8a386537088d5931e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:12:50.123253 16247 consensus_queue.cc:1035] T 6511a23cd8644897861ff82bab054511 P 32a292318dbc49b8a386537088d5931e [LEADER]: Connected to new peer: Peer: permanent_uuid: "93e20e9f1d2e4ee28d5d0891a70aa96e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:12:50.123404 16242 consensus_queue.cc:1035] T 6511a23cd8644897861ff82bab054511 P 32a292318dbc49b8a386537088d5931e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a44170426a7449ada0cd5417ec05f489" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:12:50.126138 16251 mvcc.cc:205] Tried to move back new op lower bound from 7333417656820977664 to 7333417656639123456. Current Snapshot: MvccSnapshot[applied={T|T < 7333417656820977664}]
I20260926 01:12:50.126132 16252 mvcc.cc:205] Tried to move back new op lower bound from 7333417656820977664 to 7333417656639123456. Current Snapshot: MvccSnapshot[applied={T|T < 7333417656820977664}]
I20260926 01:12:50.128026 15760 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48624:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:12:50.128139 15760 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48624:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:12:50.129037 15760 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 23f32ab05d694df7a44d5716098d6ba7: Sending DeleteTablet for 3 replicas of tablet 6511a23cd8644897861ff82bab054511
I20260926 01:12:50.129379 16154 tablet_service.cc:1514] Processing DeleteTablet for tablet 6511a23cd8644897861ff82bab054511 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:12:50 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36558
I20260926 01:12:50.129369 15886 tablet_service.cc:1514] Processing DeleteTablet for tablet 6511a23cd8644897861ff82bab054511 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:12:50 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40486
I20260926 01:12:50.129467 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 15823
I20260926 01:12:50.130005 16020 tablet_service.cc:1514] Processing DeleteTablet for tablet 6511a23cd8644897861ff82bab054511 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:12:50 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59500
I20260926 01:12:50.130221 16261 tablet_replica.cc:330] T 6511a23cd8644897861ff82bab054511 P 93e20e9f1d2e4ee28d5d0891a70aa96e: stopping tablet replica
I20260926 01:12:50.130231 16259 tablet_replica.cc:330] T 6511a23cd8644897861ff82bab054511 P 32a292318dbc49b8a386537088d5931e: stopping tablet replica
I20260926 01:12:50.130321 16259 raft_consensus.cc:2229] T 6511a23cd8644897861ff82bab054511 P 32a292318dbc49b8a386537088d5931e [term 1 LEADER]: Raft consensus shutting down.
I20260926 01:12:50.130322 16261 raft_consensus.cc:2229] T 6511a23cd8644897861ff82bab054511 P 93e20e9f1d2e4ee28d5d0891a70aa96e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:12:50.130409 16259 raft_consensus.cc:2258] T 6511a23cd8644897861ff82bab054511 P 32a292318dbc49b8a386537088d5931e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:12:50.130441 16261 raft_consensus.cc:2258] T 6511a23cd8644897861ff82bab054511 P 93e20e9f1d2e4ee28d5d0891a70aa96e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:12:50.130882 16261 ts_tablet_manager.cc:1905] T 6511a23cd8644897861ff82bab054511 P 93e20e9f1d2e4ee28d5d0891a70aa96e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:12:50.130923 16259 ts_tablet_manager.cc:1905] T 6511a23cd8644897861ff82bab054511 P 32a292318dbc49b8a386537088d5931e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:12:50.132694 16259 ts_tablet_manager.cc:1918] T 6511a23cd8644897861ff82bab054511 P 32a292318dbc49b8a386537088d5931e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:12:50.132879 16261 ts_tablet_manager.cc:1918] T 6511a23cd8644897861ff82bab054511 P 93e20e9f1d2e4ee28d5d0891a70aa96e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:12:50.132881 16259 log.cc:1196] T 6511a23cd8644897861ff82bab054511 P 32a292318dbc49b8a386537088d5931e: Deleting WAL directory at /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-2/wal/wals/6511a23cd8644897861ff82bab054511
I20260926 01:12:50.132988 16261 log.cc:1196] T 6511a23cd8644897861ff82bab054511 P 93e20e9f1d2e4ee28d5d0891a70aa96e: Deleting WAL directory at /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-1/wal/wals/6511a23cd8644897861ff82bab054511
I20260926 01:12:50.133201 16259 ts_tablet_manager.cc:1939] T 6511a23cd8644897861ff82bab054511 P 32a292318dbc49b8a386537088d5931e: Deleting consensus metadata
I20260926 01:12:50.133234 16261 ts_tablet_manager.cc:1939] T 6511a23cd8644897861ff82bab054511 P 93e20e9f1d2e4ee28d5d0891a70aa96e: Deleting consensus metadata
I20260926 01:12:50.133646 15746 catalog_manager.cc:4841] TS 32a292318dbc49b8a386537088d5931e (127.0.98.3:42549): tablet 6511a23cd8644897861ff82bab054511 (table test-table [id=4798ffee5fbe4b6191fb8e9dac55d5c5]) successfully deleted
I20260926 01:12:50.133641 15745 catalog_manager.cc:4841] TS 93e20e9f1d2e4ee28d5d0891a70aa96e (127.0.98.2:39747): tablet 6511a23cd8644897861ff82bab054511 (table test-table [id=4798ffee5fbe4b6191fb8e9dac55d5c5]) successfully deleted
W20260926 01:12:50.134966 15746 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41097 (error 108)
I20260926 01:12:50.135463 16232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:12:50.135345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41097 (local address 127.0.0.1:40494)
0926 01:12:50.135396 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0926 01:12:50.135429 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260926 01:12:50.135587 15746 catalog_manager.cc:4584] TS a44170426a7449ada0cd5417ec05f489 (127.0.98.1:41097): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6511a23cd8644897861ff82bab054511: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41097: connect: Connection refused (error 111)
I20260926 01:12:50.135864 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 15957
I20260926 01:12:50.140719 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 16091
I20260926 01:12:50.145185 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 15729
I20260926 01:13:10.150305 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45309
--webserver_interface=127.0.98.62
--webserver_port=42883
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45309
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:13:10.235292 16263 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:13:10.235638 16263 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:13:10.235688 16263 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:13:10.238111 16263 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 01:13:10.238201 16263 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 01:13:10.238224 16263 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 01:13:10.238245 16263 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:13:10.238267 16263 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:13:10.238294 16263 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:13:10.238312 16263 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:13:10.238339 16263 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:13:10.240257 16263 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45309
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45309
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=42883
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16263
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:13:10.240581 16263 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:13:10.240834 16263 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 01:13:10.251405 16263 server_base.cc:1034] running on GCE node
W20260926 01:13:10.251368 16269 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:13:10.251420 16270 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:13:10.251370 16272 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:13:10.252444 16263 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:13:10.252873 16263 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:13:10.254154 16263 hybrid_clock.cc:648] HybridClock initialized: now 1790385190254116 us; error 56 us; skew 500 ppm
Sep 26 01:13:10 dist-test-slave-7b22 krb5kdc[15716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385190, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:13:10.257072 16263 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 01:13:10.257690 16263 webserver.cc:460] Webserver started at http://127.0.98.62:42883/ using document root <none> and password file <none>
I20260926 01:13:10.257983 16263 fs_manager.cc:359] Metadata directory not provided
I20260926 01:13:10.258055 16263 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:13:10.259590 16263 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:13:10.260344 16279 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:13:10.260540 16263 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:13:10.260633 16263 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/master-0/wal
uuid: "23f32ab05d694df7a44d5716098d6ba7"
format_stamp: "Formatted at 2026-09-26 01:12:49 on dist-test-slave-7b22"
server_key: "ac8ca28dfa705c2c180098f738f866d9"
server_key_iv: "3deeff1277e5b154c414d19b3d6516b1"
server_key_version: "encryptionkey@0"
I20260926 01:13:10.261065 16263 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:13:10.290189 16263 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:13:10.293318 16263 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:13:10.293540 16263 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:13:10.297652 16263 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45309
I20260926 01:13:10.297695 16331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45309 every 8 connection(s)
I20260926 01:13:10.298121 16263 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
I20260926 01:13:10.299672 16332 sys_catalog.cc:263] Verifying existing consensus state
I20260926 01:13:10.300324 16332 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23f32ab05d694df7a44d5716098d6ba7: Bootstrap starting.
I20260926 01:13:10.306607 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 16263
I20260926 01:13:10.306939 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:41097
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:45309
--webserver_port=43205
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 01:13:10.309311 16332 log.cc:824] T 00000000000000000000000000000000 P 23f32ab05d694df7a44d5716098d6ba7: Log is configured to *not* fsync() on all Append() calls
I20260926 01:13:10.311846 16332 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23f32ab05d694df7a44d5716098d6ba7: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 01:13:10.312479 16332 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23f32ab05d694df7a44d5716098d6ba7: Bootstrap complete.
I20260926 01:13:10.314953 16332 raft_consensus.cc:348] T 00000000000000000000000000000000 P 23f32ab05d694df7a44d5716098d6ba7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23f32ab05d694df7a44d5716098d6ba7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45309 } }
I20260926 01:13:10.315219 16332 raft_consensus.cc:729] T 00000000000000000000000000000000 P 23f32ab05d694df7a44d5716098d6ba7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23f32ab05d694df7a44d5716098d6ba7, State: Initialized, Role: FOLLOWER
I20260926 01:13:10.315395 16332 consensus_queue.cc:260] T 00000000000000000000000000000000 P 23f32ab05d694df7a44d5716098d6ba7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23f32ab05d694df7a44d5716098d6ba7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45309 } }
I20260926 01:13:10.315477 16332 raft_consensus.cc:388] T 00000000000000000000000000000000 P 23f32ab05d694df7a44d5716098d6ba7 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:13:10.315537 16332 raft_consensus.cc:482] T 00000000000000000000000000000000 P 23f32ab05d694df7a44d5716098d6ba7 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:13:10.315588 16332 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 23f32ab05d694df7a44d5716098d6ba7 [term 1 FOLLOWER]: Advancing to term 2
I20260926 01:13:10.316754 16332 raft_consensus.cc:504] T 00000000000000000000000000000000 P 23f32ab05d694df7a44d5716098d6ba7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23f32ab05d694df7a44d5716098d6ba7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45309 } }
I20260926 01:13:10.316869 16332 leader_election.cc:302] T 00000000000000000000000000000000 P 23f32ab05d694df7a44d5716098d6ba7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 23f32ab05d694df7a44d5716098d6ba7; no voters:
I20260926 01:13:10.317063 16332 leader_election.cc:288] T 00000000000000000000000000000000 P 23f32ab05d694df7a44d5716098d6ba7 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 01:13:10.317116 16336 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 23f32ab05d694df7a44d5716098d6ba7 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 01:13:10.317303 16336 raft_consensus.cc:686] T 00000000000000000000000000000000 P 23f32ab05d694df7a44d5716098d6ba7 [term 2 LEADER]: Becoming Leader. State: Replica: 23f32ab05d694df7a44d5716098d6ba7, State: Running, Role: LEADER
I20260926 01:13:10.317368 16332 sys_catalog.cc:564] T 00000000000000000000000000000000 P 23f32ab05d694df7a44d5716098d6ba7 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:13:10.317459 16336 consensus_queue.cc:237] T 00000000000000000000000000000000 P 23f32ab05d694df7a44d5716098d6ba7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23f32ab05d694df7a44d5716098d6ba7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45309 } }
I20260926 01:13:10.317842 16337 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23f32ab05d694df7a44d5716098d6ba7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "23f32ab05d694df7a44d5716098d6ba7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23f32ab05d694df7a44d5716098d6ba7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45309 } } }
I20260926 01:13:10.317939 16337 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23f32ab05d694df7a44d5716098d6ba7 [sys.catalog]: This master's current role is: LEADER
I20260926 01:13:10.318260 16338 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23f32ab05d694df7a44d5716098d6ba7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 23f32ab05d694df7a44d5716098d6ba7. Latest consensus state: current_term: 2 leader_uuid: "23f32ab05d694df7a44d5716098d6ba7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23f32ab05d694df7a44d5716098d6ba7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45309 } } }
I20260926 01:13:10.318327 16338 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23f32ab05d694df7a44d5716098d6ba7 [sys.catalog]: This master's current role is: LEADER
I20260926 01:13:10.319321 16352 catalog_manager.cc:1259] Loaded cluster ID: 49a858af3ce94570a7800a2585b79192
I20260926 01:13:10.319370 16352 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 23f32ab05d694df7a44d5716098d6ba7: loading cluster ID for follower catalog manager: success
I20260926 01:13:10.320574 16352 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 23f32ab05d694df7a44d5716098d6ba7: acquiring CA information for follower catalog manager: success
I20260926 01:13:10.321026 16352 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 23f32ab05d694df7a44d5716098d6ba7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 01:13:10.321105 16343 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:13:10.321534 16343 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:13:10.321609 16343 catalog_manager.cc:1259] Loaded cluster ID: 49a858af3ce94570a7800a2585b79192
I20260926 01:13:10.321631 16343 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 01:13:10.321866 16343 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:13:10.321949 16343 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 23f32ab05d694df7a44d5716098d6ba7: Loaded TSK: 0
I20260926 01:13:10.332158 16343 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 23f32ab05d694df7a44d5716098d6ba7: Generated new TSK 1
I20260926 01:13:10.332295 16343 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 01:13:10.405758 16334 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:13:10.406075 16334 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:13:10.406117 16334 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:13:10.407543 16334 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:13:10.407606 16334 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:13:10.407649 16334 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 01:13:10.409466 16334 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:41097
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=43205
--tserver_master_addrs=127.0.98.62:45309
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16334
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:13:10.410107 16334 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:13:10.410485 16334 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:13:10.417925 16362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:13:10.417954 16360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:13:10.417928 16359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:13:10.418807 16334 server_base.cc:1034] running on GCE node
I20260926 01:13:10.419081 16334 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:13:10.419428 16334 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:13:10.420629 16334 hybrid_clock.cc:648] HybridClock initialized: now 1790385190420602 us; error 43 us; skew 500 ppm
Sep 26 01:13:10 dist-test-slave-7b22 krb5kdc[15716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385190, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:13:10.424235 16334 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 01:13:10.424911 16334 webserver.cc:460] Webserver started at http://127.0.98.1:43205/ using document root <none> and password file <none>
I20260926 01:13:10.425211 16334 fs_manager.cc:359] Metadata directory not provided
I20260926 01:13:10.425305 16334 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:13:10.427263 16334 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 01:13:10.428242 16369 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:13:10.428508 16334 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 01:13:10.428617 16334 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-0/wal
uuid: "a44170426a7449ada0cd5417ec05f489"
format_stamp: "Formatted at 2026-09-26 01:12:49 on dist-test-slave-7b22"
server_key: "df9ded4ee0a9fbc082989a9db96fa076"
server_key_iv: "79f3b89faba1690855051b5d290bf1c4"
server_key_version: "encryptionkey@0"
I20260926 01:13:10.429134 16334 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:13:10.450397 16334 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:13:10.452911 16334 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:13:10.453040 16334 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:13:10.453306 16334 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:13:10.453847 16376 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 01:13:10.455425 16334 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 01:13:10.455480 16334 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260926 01:13:10.455508 16334 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 01:13:10.456197 16334 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 01:13:10.456254 16334 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260926 01:13:10.456324 16376 tablet_bootstrap.cc:492] T 6511a23cd8644897861ff82bab054511 P a44170426a7449ada0cd5417ec05f489: Bootstrap starting.
I20260926 01:13:10.465641 16334 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41097
I20260926 01:13:10.465883 16483 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41097 every 8 connection(s)
I20260926 01:13:10.466270 16334 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-0/data/info.pb
I20260926 01:13:10.469458 16376 log.cc:824] T 6511a23cd8644897861ff82bab054511 P a44170426a7449ada0cd5417ec05f489: Log is configured to *not* fsync() on all Append() calls
I20260926 01:13:10.472684 16376 tablet_bootstrap.cc:492] T 6511a23cd8644897861ff82bab054511 P a44170426a7449ada0cd5417ec05f489: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 01:13:10.472980 16376 tablet_bootstrap.cc:492] T 6511a23cd8644897861ff82bab054511 P a44170426a7449ada0cd5417ec05f489: Bootstrap complete.
I20260926 01:13:10.473937 16376 ts_tablet_manager.cc:1397] T 6511a23cd8644897861ff82bab054511 P a44170426a7449ada0cd5417ec05f489: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.013s
I20260926 01:13:10.473949 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 16334
I20260926 01:13:10.474239 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:39747
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:45309
--webserver_port=39883
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 01:13:10.475598 16376 raft_consensus.cc:348] T 6511a23cd8644897861ff82bab054511 P a44170426a7449ada0cd5417ec05f489 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44170426a7449ada0cd5417ec05f489" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41097 } } peers { permanent_uuid: "93e20e9f1d2e4ee28d5d0891a70aa96e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39747 } } peers { permanent_uuid: "32a292318dbc49b8a386537088d5931e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42549 } }
I20260926 01:13:10.475996 16376 raft_consensus.cc:729] T 6511a23cd8644897861ff82bab054511 P a44170426a7449ada0cd5417ec05f489 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a44170426a7449ada0cd5417ec05f489, State: Initialized, Role: FOLLOWER
I20260926 01:13:10.476174 16376 consensus_queue.cc:260] T 6511a23cd8644897861ff82bab054511 P a44170426a7449ada0cd5417ec05f489 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44170426a7449ada0cd5417ec05f489" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41097 } } peers { permanent_uuid: "93e20e9f1d2e4ee28d5d0891a70aa96e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39747 } } peers { permanent_uuid: "32a292318dbc49b8a386537088d5931e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42549 } }
I20260926 01:13:10.476490 16376 ts_tablet_manager.cc:1428] T 6511a23cd8644897861ff82bab054511 P a44170426a7449ada0cd5417ec05f489: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
Sep 26 01:13:10 dist-test-slave-7b22 krb5kdc[15716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385190, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:13:10.483489 16487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:10.467296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60917 (local address 127.0.98.62:45309)
0926 01:13:10.467537 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0926 01:13:10.467547 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 01:13:10.470176 (+ 2629us) server_negotiation.cc:407] Connection header received
0926 01:13:10.470823 (+ 647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:13:10.470831 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:13:10.470894 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:13:10.471056 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:13:10.471918 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:10.473315 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:10.475189 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:10.475836 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:10.475938 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:10.479055 (+ 3117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:13:10.479060 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:13:10.479061 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:13:10.479076 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:13:10.481169 (+ 2093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:13:10.481755 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:13:10.481760 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:13:10.481761 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:13:10.481929 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:13:10.482244 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:13:10.482245 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:13:10.482245 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:13:10.482578 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:13:10.482652 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 01:13:10.483087 (+ 435us) server_negotiation.cc:299] Negotiation successful
0926 01:13:10.483262 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":140,"threads_started":1}
I20260926 01:13:10.484395 16486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:10.467157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45309 (local address 127.0.98.1:60917)
0926 01:13:10.469966 (+ 2809us) negotiation.cc:107] Waiting for socket to connect
0926 01:13:10.470014 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 01:13:10.470558 (+ 544us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:13:10.471116 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:13:10.471121 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:13:10.471497 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:13:10.471765 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:13:10.471772 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:13:10.473479 (+ 1707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:13:10.473480 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:13:10.475047 (+ 1567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:13:10.475068 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:13:10.476071 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:13:10.476072 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:13:10.476227 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:10.476895 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:13:10.476905 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:13:10.478907 (+ 2002us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:13:10.481290 (+ 2383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:13:10.481301 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:13:10.481304 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:13:10.481644 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:13:10.482013 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:13:10.482013 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:13:10.482014 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:13:10.482161 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:13:10.482691 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:13:10.482700 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:13:10.482880 (+ 180us) client_negotiation.cc:769] Sending connection context
0926 01:13:10.483708 (+ 828us) client_negotiation.cc:240] Negotiation successful
0926 01:13:10.484180 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2745,"thread_start_us":99,"threads_started":1}
I20260926 01:13:10.485423 16484 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45309
I20260926 01:13:10.485569 16484 heartbeater.cc:464] Registering TS with master...
I20260926 01:13:10.485791 16484 heartbeater.cc:511] Master 127.0.98.62:45309 requested a full tablet report, sending...
I20260926 01:13:10.486690 16296 ts_manager.cc:194] Registered new tserver with Master: a44170426a7449ada0cd5417ec05f489 (127.0.98.1:41097)
I20260926 01:13:10.487859 16296 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:60917
I20260926 01:13:10.488902 16484 heartbeater.cc:503] Master 127.0.98.62:45309 was elected leader, sending a full tablet report...
Sep 26 01:13:10 dist-test-slave-7b22 krb5kdc[15716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385190, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 01:13:10.498107 16492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:10.487400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44364 (local address 127.0.98.1:41097)
0926 01:13:10.487590 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 01:13:10.487595 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 01:13:10.487628 (+ 33us) server_negotiation.cc:407] Connection header received
0926 01:13:10.487734 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:13:10.487736 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:13:10.487750 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:13:10.487840 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:13:10.488341 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:10.489075 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:10.490349 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:10.490662 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:10.490728 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:10.493739 (+ 3011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:13:10.493760 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:13:10.493760 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:13:10.493773 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:13:10.496009 (+ 2236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:13:10.496888 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:13:10.496893 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:13:10.496893 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:13:10.496975 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:13:10.497236 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:13:10.497237 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:13:10.497237 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:13:10.497462 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:13:10.497555 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 01:13:10.497812 (+ 257us) server_negotiation.cc:299] Negotiation successful
0926 01:13:10.497970 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":95,"threads_started":1}
I20260926 01:13:10.498226 16491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:10.487304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41097 (local address 127.0.0.1:44364)
0926 01:13:10.487544 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0926 01:13:10.487578 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 01:13:10.487647 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:13:10.487845 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:13:10.487848 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:13:10.488199 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:13:10.488264 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:13:10.488266 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:13:10.489196 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:13:10.489197 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:13:10.490204 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:13:10.490207 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:13:10.490734 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:13:10.490737 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:13:10.490812 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:10.491344 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:13:10.491349 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:13:10.493454 (+ 2105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:13:10.496157 (+ 2703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:13:10.496171 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:13:10.496174 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:13:10.496749 (+ 575us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:13:10.497043 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:13:10.497044 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:13:10.497044 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:13:10.497146 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:13:10.497547 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:13:10.497554 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:13:10.497650 (+ 96us) client_negotiation.cc:769] Sending connection context
0926 01:13:10.497781 (+ 131us) client_negotiation.cc:240] Negotiation successful
0926 01:13:10.498072 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":111,"threads_started":1}
I20260926 01:13:10.498936 16418 tablet_service.cc:1514] Processing DeleteTablet for tablet 6511a23cd8644897861ff82bab054511 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:12:50 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44364
I20260926 01:13:10.500504 16493 tablet_replica.cc:330] stopping tablet replica
I20260926 01:13:10.500674 16493 raft_consensus.cc:2229] T 6511a23cd8644897861ff82bab054511 P a44170426a7449ada0cd5417ec05f489 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:13:10.500777 16493 raft_consensus.cc:2258] T 6511a23cd8644897861ff82bab054511 P a44170426a7449ada0cd5417ec05f489 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:13:10.501185 16493 ts_tablet_manager.cc:1905] T 6511a23cd8644897861ff82bab054511 P a44170426a7449ada0cd5417ec05f489: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:13:10.504163 16493 ts_tablet_manager.cc:1918] T 6511a23cd8644897861ff82bab054511 P a44170426a7449ada0cd5417ec05f489: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:13:10.504254 16493 log.cc:1196] T 6511a23cd8644897861ff82bab054511 P a44170426a7449ada0cd5417ec05f489: Deleting WAL directory at /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-0/wal/wals/6511a23cd8644897861ff82bab054511
I20260926 01:13:10.504480 16493 ts_tablet_manager.cc:1939] T 6511a23cd8644897861ff82bab054511 P a44170426a7449ada0cd5417ec05f489: Deleting consensus metadata
I20260926 01:13:10.505097 16281 catalog_manager.cc:4841] TS a44170426a7449ada0cd5417ec05f489 (127.0.98.1:41097): tablet 6511a23cd8644897861ff82bab054511 (table test-table [id=4798ffee5fbe4b6191fb8e9dac55d5c5]) successfully deleted
W20260926 01:13:10.569339 16489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:13:10.569590 16489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:13:10.569617 16489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:13:10.570933 16489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:13:10.570979 16489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:13:10.571009 16489 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 01:13:10.572587 16489 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:39747
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=39883
--tserver_master_addrs=127.0.98.62:45309
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16489
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:13:10.572809 16489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:13:10.573015 16489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:13:10.579638 16498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:13:10.579656 16499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:13:10.579701 16489 server_base.cc:1034] running on GCE node
W20260926 01:13:10.579648 16501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:13:10.580193 16489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:13:10.580505 16489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:13:10.581677 16489 hybrid_clock.cc:648] HybridClock initialized: now 1790385190581630 us; error 65 us; skew 500 ppm
Sep 26 01:13:10 dist-test-slave-7b22 krb5kdc[15716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385190, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:13:10.584724 16489 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 01:13:10.585264 16489 webserver.cc:460] Webserver started at http://127.0.98.2:39883/ using document root <none> and password file <none>
I20260926 01:13:10.585493 16489 fs_manager.cc:359] Metadata directory not provided
I20260926 01:13:10.585559 16489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:13:10.587054 16489 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:13:10.587749 16508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:13:10.588002 16489 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:13:10.588105 16489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-1/wal
uuid: "93e20e9f1d2e4ee28d5d0891a70aa96e"
format_stamp: "Formatted at 2026-09-26 01:12:49 on dist-test-slave-7b22"
server_key: "792d916d3335ac55de814f8a8c732b05"
server_key_iv: "a3b5ba14f34ab0797dc72053300ca2e5"
server_key_version: "encryptionkey@0"
I20260926 01:13:10.588541 16489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:13:10.595753 16489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:13:10.598611 16489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:13:10.598762 16489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:13:10.599000 16489 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:13:10.599315 16489 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:13:10.599347 16489 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:13:10.599390 16489 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:13:10.599413 16489 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:13:10.607576 16489 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39747
I20260926 01:13:10.607712 16621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39747 every 8 connection(s)
I20260926 01:13:10.608127 16489 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-1/data/info.pb
I20260926 01:13:10.611796 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 16489
I20260926 01:13:10.612139 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:42549
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:45309
--webserver_port=43609
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 01:13:10 dist-test-slave-7b22 krb5kdc[15716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385190, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:13:10.621681 16487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:10.608963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39183 (local address 127.0.98.62:45309)
0926 01:13:10.609051 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0926 01:13:10.609054 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:13:10.609503 (+ 449us) server_negotiation.cc:407] Connection header received
0926 01:13:10.610287 (+ 784us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:13:10.610288 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:13:10.610311 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:13:10.610380 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:13:10.611725 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:10.612458 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:10.613761 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:10.614024 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:10.614090 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:10.617454 (+ 3364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:13:10.617456 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:13:10.617457 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:13:10.617467 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:13:10.619523 (+ 2056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:13:10.620204 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:13:10.620206 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:13:10.620206 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:13:10.620272 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:13:10.620688 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:13:10.620691 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:13:10.620692 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:13:10.620900 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:13:10.620997 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 01:13:10.621483 (+ 486us) server_negotiation.cc:299] Negotiation successful
0926 01:13:10.621592 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 01:13:10.621954 16624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:10.608983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45309 (local address 127.0.98.2:39183)
0926 01:13:10.609316 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0926 01:13:10.609366 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 01:13:10.609970 (+ 604us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:13:10.610499 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:13:10.610503 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:13:10.611337 (+ 834us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:13:10.611559 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:13:10.611564 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:13:10.612578 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:13:10.612579 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:13:10.613659 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:13:10.613663 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:13:10.614106 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:13:10.614107 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:13:10.614286 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:10.615081 (+ 795us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:13:10.615091 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:13:10.617204 (+ 2113us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:13:10.619647 (+ 2443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:13:10.619665 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:13:10.619669 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:13:10.620094 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:13:10.620349 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:13:10.620350 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:13:10.620350 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:13:10.620513 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:13:10.621005 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:13:10.621013 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:13:10.621255 (+ 242us) client_negotiation.cc:769] Sending connection context
0926 01:13:10.621410 (+ 155us) client_negotiation.cc:240] Negotiation successful
0926 01:13:10.621624 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":124,"threads_started":1}
I20260926 01:13:10.622941 16622 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45309
I20260926 01:13:10.623072 16622 heartbeater.cc:464] Registering TS with master...
I20260926 01:13:10.623337 16622 heartbeater.cc:511] Master 127.0.98.62:45309 requested a full tablet report, sending...
I20260926 01:13:10.623899 16296 ts_manager.cc:194] Registered new tserver with Master: 93e20e9f1d2e4ee28d5d0891a70aa96e (127.0.98.2:39747)
I20260926 01:13:10.624365 16296 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:39183
W20260926 01:13:10.707906 16625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:13:10.708158 16625 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:13:10.708185 16625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:13:10.710860 16625 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:13:10.710955 16625 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:13:10.711007 16625 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 01:13:10.713155 16625 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:42549
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=43609
--tserver_master_addrs=127.0.98.62:45309
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16625
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:13:10.713423 16625 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:13:10.713634 16625 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:13:10.719555 16631 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:13:10.719558 16633 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:13:10.719650 16630 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:13:10.720145 16625 server_base.cc:1034] running on GCE node
I20260926 01:13:10.720407 16625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:13:10.720688 16625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:13:10.721933 16625 hybrid_clock.cc:648] HybridClock initialized: now 1790385190721890 us; error 82 us; skew 500 ppm
Sep 26 01:13:10 dist-test-slave-7b22 krb5kdc[15716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385190, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:13:10.726029 16625 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 01:13:10.726543 16625 webserver.cc:460] Webserver started at http://127.0.98.3:43609/ using document root <none> and password file <none>
I20260926 01:13:10.726804 16625 fs_manager.cc:359] Metadata directory not provided
I20260926 01:13:10.726873 16625 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:13:10.728438 16625 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:13:10.729557 16640 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:13:10.729988 16625 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:13:10.730154 16625 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-2/wal
uuid: "32a292318dbc49b8a386537088d5931e"
format_stamp: "Formatted at 2026-09-26 01:12:49 on dist-test-slave-7b22"
server_key: "40190f19815e04d52f09f97fef649f8e"
server_key_iv: "f9942246e0fa6846e7aeeb7ee71e1f3e"
server_key_version: "encryptionkey@0"
I20260926 01:13:10.730733 16625 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:13:10.739660 16625 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:13:10.742357 16625 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:13:10.742519 16625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:13:10.742765 16625 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:13:10.743129 16625 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:13:10.743163 16625 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:13:10.743196 16625 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:13:10.743227 16625 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:13:10.749272 16625 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42549
I20260926 01:13:10.749360 16753 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42549 every 8 connection(s)
I20260926 01:13:10.749699 16625 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-2/data/info.pb
Sep 26 01:13:10 dist-test-slave-7b22 krb5kdc[15716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385190, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:13:10.759428 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 16625
I20260926 01:13:10.760811 16487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:10.750531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59415 (local address 127.0.98.62:45309)
0926 01:13:10.750615 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0926 01:13:10.750618 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:13:10.750899 (+ 281us) server_negotiation.cc:407] Connection header received
0926 01:13:10.751551 (+ 652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:13:10.751553 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:13:10.751564 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:13:10.751626 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:13:10.753141 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:10.753664 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:10.754895 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:10.755092 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:10.755161 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:10.757890 (+ 2729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:13:10.757892 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:13:10.757893 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:13:10.757901 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:13:10.759209 (+ 1308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:13:10.759744 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:13:10.759745 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:13:10.759746 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:13:10.759811 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:13:10.760090 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:13:10.760091 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:13:10.760092 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:13:10.760257 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:13:10.760325 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 01:13:10.760632 (+ 307us) server_negotiation.cc:299] Negotiation successful
0926 01:13:10.760718 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 01:13:10.761042 16756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:10.750509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45309 (local address 127.0.98.3:59415)
0926 01:13:10.750748 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0926 01:13:10.750802 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 01:13:10.751388 (+ 586us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:13:10.751846 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:13:10.751853 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:13:10.752805 (+ 952us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:13:10.753024 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:13:10.753032 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:13:10.753800 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:13:10.753801 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:13:10.754796 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:13:10.754799 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:13:10.755196 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:13:10.755197 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:13:10.755325 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:10.755971 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:13:10.755981 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:13:10.757670 (+ 1689us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:13:10.759329 (+ 1659us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:13:10.759337 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:13:10.759339 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:13:10.759665 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:13:10.759885 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:13:10.759886 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:13:10.759886 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:13:10.760023 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:13:10.760349 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:13:10.760356 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:13:10.760505 (+ 149us) client_negotiation.cc:769] Sending connection context
0926 01:13:10.760604 (+ 99us) client_negotiation.cc:240] Negotiation successful
0926 01:13:10.760788 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":91,"threads_started":1}
I20260926 01:13:10.761497 16754 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45309
I20260926 01:13:10.761595 16754 heartbeater.cc:464] Registering TS with master...
I20260926 01:13:10.761777 16754 heartbeater.cc:511] Master 127.0.98.62:45309 requested a full tablet report, sending...
I20260926 01:13:10.762207 16296 ts_manager.cc:194] Registered new tserver with Master: 32a292318dbc49b8a386537088d5931e (127.0.98.3:42549)
I20260926 01:13:10.762563 16296 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:59415
I20260926 01:13:10.766634 16758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:10.759830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35572 (local address 127.0.98.62:45309)
0926 01:13:10.759982 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0926 01:13:10.759984 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:13:10.760108 (+ 124us) server_negotiation.cc:407] Connection header received
0926 01:13:10.760183 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:13:10.760183 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:13:10.760194 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:13:10.760279 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:13:10.760886 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:10.761620 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:10.762537 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:10.762794 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:10.762844 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:10.764074 (+ 1230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:13:10.764076 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:13:10.764076 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:13:10.764083 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:13:10.765544 (+ 1461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:13:10.765906 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:13:10.765907 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:13:10.765908 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:13:10.765966 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:13:10.766127 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:13:10.766128 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:13:10.766128 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:13:10.766248 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:13:10.766423 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0926 01:13:10.766458 (+ 35us) server_negotiation.cc:299] Negotiation successful
0926 01:13:10.766538 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":89,"threads_started":1}
I20260926 01:13:10.767532 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 01:13:10.770675 16758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:10.768015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35588 (local address 127.0.98.62:45309)
0926 01:13:10.768144 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0926 01:13:10.768146 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:13:10.768157 (+ 11us) server_negotiation.cc:407] Connection header received
0926 01:13:10.768231 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:13:10.768232 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:13:10.768263 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:13:10.768377 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:13:10.768584 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:10.769085 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:10.769940 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:10.770156 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:10.770210 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:10.770394 (+ 184us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:13:10.770452 (+ 58us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0926 01:13:10.770603 (+ 151us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35588: token expired
Metrics: {"server-negotiator.queue_time_us":105}
W20260926 01:13:10.770754 16758 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35588: token expired
W20260926 01:13:10.770911 16224 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.98.62:45309)
I20260926 01:13:10.776741 16758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:10.771169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35600 (local address 127.0.98.62:45309)
0926 01:13:10.771226 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 01:13:10.771227 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 01:13:10.771236 (+ 9us) server_negotiation.cc:407] Connection header received
0926 01:13:10.771328 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:13:10.771329 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:13:10.771338 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:13:10.771383 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:13:10.771838 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:10.772293 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:10.772942 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:10.773123 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:10.773176 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:10.774360 (+ 1184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:13:10.774362 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:13:10.774362 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:13:10.774370 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:13:10.775728 (+ 1358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:13:10.776046 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:13:10.776047 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:13:10.776048 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:13:10.776098 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:13:10.776255 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:13:10.776255 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:13:10.776255 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:13:10.776368 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:13:10.776454 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 01:13:10.776569 (+ 115us) server_negotiation.cc:299] Negotiation successful
0926 01:13:10.776649 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 01:13:10.778007 16296 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35600:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:13:10.779680 16296 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:13:10.781986 16418 tablet_service.cc:1467] Processing CreateTablet for tablet 98418cdfbca94d9ba37eb367788fa8ee (DEFAULT_TABLE table=test-table [id=c47dbfbef52741ef9f4fc7df43073f58]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:13:10.782291 16418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98418cdfbca94d9ba37eb367788fa8ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:13:10.784021 16376 tablet_bootstrap.cc:492] T 98418cdfbca94d9ba37eb367788fa8ee P a44170426a7449ada0cd5417ec05f489: Bootstrap starting.
I20260926 01:13:10.784521 16376 tablet_bootstrap.cc:654] T 98418cdfbca94d9ba37eb367788fa8ee P a44170426a7449ada0cd5417ec05f489: Neither blocks nor log segments found. Creating new log.
I20260926 01:13:10.785498 16376 tablet_bootstrap.cc:492] T 98418cdfbca94d9ba37eb367788fa8ee P a44170426a7449ada0cd5417ec05f489: No bootstrap required, opened a new log
I20260926 01:13:10.785561 16376 ts_tablet_manager.cc:1397] T 98418cdfbca94d9ba37eb367788fa8ee P a44170426a7449ada0cd5417ec05f489: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 01:13:10.785698 16491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:10.781489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39747 (local address 127.0.0.1:58278)
0926 01:13:10.781546 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0926 01:13:10.781558 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 01:13:10.781607 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:13:10.782365 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:13:10.782367 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:13:10.782369 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:13:10.782417 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:13:10.782418 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:13:10.783628 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:13:10.783629 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:13:10.784554 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:13:10.784557 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:13:10.785375 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:13:10.785377 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:13:10.785457 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:10.785501 (+ 44us) client_negotiation.cc:769] Sending connection context
0926 01:13:10.785535 (+ 34us) client_negotiation.cc:240] Negotiation successful
0926 01:13:10.785591 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260926 01:13:10.785720 16376 raft_consensus.cc:348] T 98418cdfbca94d9ba37eb367788fa8ee P a44170426a7449ada0cd5417ec05f489 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e20e9f1d2e4ee28d5d0891a70aa96e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39747 } } peers { permanent_uuid: "32a292318dbc49b8a386537088d5931e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42549 } } peers { permanent_uuid: "a44170426a7449ada0cd5417ec05f489" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41097 } }
I20260926 01:13:10.785810 16376 raft_consensus.cc:374] T 98418cdfbca94d9ba37eb367788fa8ee P a44170426a7449ada0cd5417ec05f489 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:13:10.785897 16376 raft_consensus.cc:729] T 98418cdfbca94d9ba37eb367788fa8ee P a44170426a7449ada0cd5417ec05f489 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a44170426a7449ada0cd5417ec05f489, State: Initialized, Role: FOLLOWER
I20260926 01:13:10.785928 16762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:10.781658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58278 (local address 127.0.98.2:39747)
0926 01:13:10.782194 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0926 01:13:10.782198 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 01:13:10.782207 (+ 9us) server_negotiation.cc:407] Connection header received
0926 01:13:10.782243 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:13:10.782245 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:13:10.782260 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:13:10.782356 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:13:10.782512 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:10.783503 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:10.784661 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:10.785275 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:10.785679 (+ 404us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:10.785743 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 01:13:10.785771 (+ 28us) server_negotiation.cc:299] Negotiation successful
0926 01:13:10.785850 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":502,"thread_start_us":90,"threads_started":1}
I20260926 01:13:10.785948 16376 consensus_queue.cc:260] T 98418cdfbca94d9ba37eb367788fa8ee P a44170426a7449ada0cd5417ec05f489 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e20e9f1d2e4ee28d5d0891a70aa96e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39747 } } peers { permanent_uuid: "32a292318dbc49b8a386537088d5931e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42549 } } peers { permanent_uuid: "a44170426a7449ada0cd5417ec05f489" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41097 } }
I20260926 01:13:10.786036 16376 ts_tablet_manager.cc:1428] T 98418cdfbca94d9ba37eb367788fa8ee P a44170426a7449ada0cd5417ec05f489: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260926 01:13:10.786033 16761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:10.781665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39924 (local address 127.0.98.3:42549)
0926 01:13:10.781927 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0926 01:13:10.781932 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 01:13:10.781941 (+ 9us) server_negotiation.cc:407] Connection header received
0926 01:13:10.781975 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:13:10.781976 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:13:10.781988 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:13:10.782058 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:13:10.782217 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:10.783267 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:10.784548 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:10.785358 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:10.785432 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:10.785497 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 01:13:10.785914 (+ 417us) server_negotiation.cc:299] Negotiation successful
0926 01:13:10.785956 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":82,"threads_started":1}
I20260926 01:13:10.786165 16760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:10.781489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42549 (local address 127.0.0.1:39924)
0926 01:13:10.781741 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0926 01:13:10.781751 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 01:13:10.781792 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:13:10.782079 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:13:10.782080 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:13:10.782083 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:13:10.782142 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:13:10.782144 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:13:10.783342 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:13:10.783343 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:13:10.784420 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:13:10.784424 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:13:10.785451 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:13:10.785452 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:13:10.785513 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:10.785517 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 01:13:10.785563 (+ 46us) client_negotiation.cc:240] Negotiation successful
0926 01:13:10.785612 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"mutex_wait_us":61,"thread_start_us":73,"threads_started":1}
I20260926 01:13:10.786659 16556 tablet_service.cc:1467] Processing CreateTablet for tablet 98418cdfbca94d9ba37eb367788fa8ee (DEFAULT_TABLE table=test-table [id=c47dbfbef52741ef9f4fc7df43073f58]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:13:10.786810 16688 tablet_service.cc:1467] Processing CreateTablet for tablet 98418cdfbca94d9ba37eb367788fa8ee (DEFAULT_TABLE table=test-table [id=c47dbfbef52741ef9f4fc7df43073f58]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:13:10.787043 16556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98418cdfbca94d9ba37eb367788fa8ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:13:10.787133 16688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98418cdfbca94d9ba37eb367788fa8ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:13:10.789780 16764 tablet_bootstrap.cc:492] T 98418cdfbca94d9ba37eb367788fa8ee P 32a292318dbc49b8a386537088d5931e: Bootstrap starting.
I20260926 01:13:10.790172 16765 tablet_bootstrap.cc:492] T 98418cdfbca94d9ba37eb367788fa8ee P 93e20e9f1d2e4ee28d5d0891a70aa96e: Bootstrap starting.
I20260926 01:13:10.790657 16764 tablet_bootstrap.cc:654] T 98418cdfbca94d9ba37eb367788fa8ee P 32a292318dbc49b8a386537088d5931e: Neither blocks nor log segments found. Creating new log.
I20260926 01:13:10.790825 16765 tablet_bootstrap.cc:654] T 98418cdfbca94d9ba37eb367788fa8ee P 93e20e9f1d2e4ee28d5d0891a70aa96e: Neither blocks nor log segments found. Creating new log.
I20260926 01:13:10.791311 16764 log.cc:824] T 98418cdfbca94d9ba37eb367788fa8ee P 32a292318dbc49b8a386537088d5931e: Log is configured to *not* fsync() on all Append() calls
I20260926 01:13:10.791401 16765 log.cc:824] T 98418cdfbca94d9ba37eb367788fa8ee P 93e20e9f1d2e4ee28d5d0891a70aa96e: Log is configured to *not* fsync() on all Append() calls
I20260926 01:13:10.792225 16764 tablet_bootstrap.cc:492] T 98418cdfbca94d9ba37eb367788fa8ee P 32a292318dbc49b8a386537088d5931e: No bootstrap required, opened a new log
I20260926 01:13:10.792263 16765 tablet_bootstrap.cc:492] T 98418cdfbca94d9ba37eb367788fa8ee P 93e20e9f1d2e4ee28d5d0891a70aa96e: No bootstrap required, opened a new log
I20260926 01:13:10.792301 16764 ts_tablet_manager.cc:1397] T 98418cdfbca94d9ba37eb367788fa8ee P 32a292318dbc49b8a386537088d5931e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 01:13:10.792315 16765 ts_tablet_manager.cc:1397] T 98418cdfbca94d9ba37eb367788fa8ee P 93e20e9f1d2e4ee28d5d0891a70aa96e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 01:13:10.794343 16764 raft_consensus.cc:348] T 98418cdfbca94d9ba37eb367788fa8ee P 32a292318dbc49b8a386537088d5931e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e20e9f1d2e4ee28d5d0891a70aa96e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39747 } } peers { permanent_uuid: "32a292318dbc49b8a386537088d5931e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42549 } } peers { permanent_uuid: "a44170426a7449ada0cd5417ec05f489" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41097 } }
I20260926 01:13:10.794343 16765 raft_consensus.cc:348] T 98418cdfbca94d9ba37eb367788fa8ee P 93e20e9f1d2e4ee28d5d0891a70aa96e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e20e9f1d2e4ee28d5d0891a70aa96e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39747 } } peers { permanent_uuid: "32a292318dbc49b8a386537088d5931e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42549 } } peers { permanent_uuid: "a44170426a7449ada0cd5417ec05f489" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41097 } }
I20260926 01:13:10.794550 16764 raft_consensus.cc:374] T 98418cdfbca94d9ba37eb367788fa8ee P 32a292318dbc49b8a386537088d5931e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:13:10.794561 16765 raft_consensus.cc:374] T 98418cdfbca94d9ba37eb367788fa8ee P 93e20e9f1d2e4ee28d5d0891a70aa96e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:13:10.794592 16764 raft_consensus.cc:729] T 98418cdfbca94d9ba37eb367788fa8ee P 32a292318dbc49b8a386537088d5931e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32a292318dbc49b8a386537088d5931e, State: Initialized, Role: FOLLOWER
I20260926 01:13:10.794592 16765 raft_consensus.cc:729] T 98418cdfbca94d9ba37eb367788fa8ee P 93e20e9f1d2e4ee28d5d0891a70aa96e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93e20e9f1d2e4ee28d5d0891a70aa96e, State: Initialized, Role: FOLLOWER
I20260926 01:13:10.794750 16764 consensus_queue.cc:260] T 98418cdfbca94d9ba37eb367788fa8ee P 32a292318dbc49b8a386537088d5931e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e20e9f1d2e4ee28d5d0891a70aa96e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39747 } } peers { permanent_uuid: "32a292318dbc49b8a386537088d5931e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42549 } } peers { permanent_uuid: "a44170426a7449ada0cd5417ec05f489" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41097 } }
I20260926 01:13:10.794750 16765 consensus_queue.cc:260] T 98418cdfbca94d9ba37eb367788fa8ee P 93e20e9f1d2e4ee28d5d0891a70aa96e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e20e9f1d2e4ee28d5d0891a70aa96e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39747 } } peers { permanent_uuid: "32a292318dbc49b8a386537088d5931e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42549 } } peers { permanent_uuid: "a44170426a7449ada0cd5417ec05f489" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41097 } }
I20260926 01:13:10.795076 16764 ts_tablet_manager.cc:1428] T 98418cdfbca94d9ba37eb367788fa8ee P 32a292318dbc49b8a386537088d5931e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 01:13:10.795076 16765 ts_tablet_manager.cc:1428] T 98418cdfbca94d9ba37eb367788fa8ee P 93e20e9f1d2e4ee28d5d0891a70aa96e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 01:13:10.795096 16622 heartbeater.cc:503] Master 127.0.98.62:45309 was elected leader, sending a full tablet report...
I20260926 01:13:10.795275 16754 heartbeater.cc:503] Master 127.0.98.62:45309 was elected leader, sending a full tablet report...
W20260926 01:13:10.858794 16623 tablet.cc:2369] T 98418cdfbca94d9ba37eb367788fa8ee P 93e20e9f1d2e4ee28d5d0891a70aa96e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 01:13:10.970286 16485 tablet.cc:2369] T 98418cdfbca94d9ba37eb367788fa8ee P a44170426a7449ada0cd5417ec05f489: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 01:13:10.970907 16490 raft_consensus.cc:482] T 98418cdfbca94d9ba37eb367788fa8ee P a44170426a7449ada0cd5417ec05f489 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:13:10.971014 16490 raft_consensus.cc:504] T 98418cdfbca94d9ba37eb367788fa8ee P a44170426a7449ada0cd5417ec05f489 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e20e9f1d2e4ee28d5d0891a70aa96e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39747 } } peers { permanent_uuid: "32a292318dbc49b8a386537088d5931e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42549 } } peers { permanent_uuid: "a44170426a7449ada0cd5417ec05f489" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41097 } }
I20260926 01:13:10.971385 16490 leader_election.cc:288] T 98418cdfbca94d9ba37eb367788fa8ee P a44170426a7449ada0cd5417ec05f489 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 93e20e9f1d2e4ee28d5d0891a70aa96e (127.0.98.2:39747), 32a292318dbc49b8a386537088d5931e (127.0.98.3:42549)
I20260926 01:13:10.975531 16762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:10.971657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60179 (local address 127.0.98.2:39747)
0926 01:13:10.971713 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 01:13:10.971715 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:13:10.971934 (+ 219us) server_negotiation.cc:407] Connection header received
0926 01:13:10.971965 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:13:10.971966 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:13:10.971977 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:13:10.972018 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:13:10.972133 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:10.972850 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:10.974412 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:10.975027 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:10.975105 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:10.975117 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0926 01:13:10.975398 (+ 281us) server_negotiation.cc:299] Negotiation successful
0926 01:13:10.975453 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260926 01:13:10.975517 16486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:10.971593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42549 (local address 127.0.98.1:56039)
0926 01:13:10.971657 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0926 01:13:10.971670 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 01:13:10.971730 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:13:10.971926 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:13:10.971927 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:13:10.971929 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:13:10.972007 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:13:10.972009 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:13:10.972942 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:13:10.972942 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:13:10.974299 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:13:10.974302 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:13:10.975165 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:13:10.975167 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:13:10.975262 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:10.975285 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 01:13:10.975367 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 01:13:10.975407 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":53}
I20260926 01:13:10.975531 16761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:10.971679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56039 (local address 127.0.98.3:42549)
0926 01:13:10.971738 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 01:13:10.971740 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:13:10.971750 (+ 10us) server_negotiation.cc:407] Connection header received
0926 01:13:10.971816 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:13:10.971817 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:13:10.971829 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:13:10.971904 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:13:10.972075 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:10.972838 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:10.974421 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:10.975027 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:10.975105 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:10.975117 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0926 01:13:10.975405 (+ 288us) server_negotiation.cc:299] Negotiation successful
0926 01:13:10.975458 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 01:13:10.975524 16770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:10.971594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39747 (local address 127.0.98.1:60179)
0926 01:13:10.971871 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0926 01:13:10.971878 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0926 01:13:10.971925 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:13:10.972023 (+ 98us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:13:10.972023 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:13:10.972024 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:13:10.972079 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:13:10.972080 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:13:10.972938 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:13:10.972939 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:13:10.974299 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:13:10.974302 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:13:10.975186 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:13:10.975187 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:13:10.975254 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:10.975285 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 01:13:10.975358 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 01:13:10.975407 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"mutex_wait_us":241,"thread_start_us":74,"threads_started":1}
I20260926 01:13:10.975922 16708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98418cdfbca94d9ba37eb367788fa8ee" candidate_uuid: "a44170426a7449ada0cd5417ec05f489" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32a292318dbc49b8a386537088d5931e" is_pre_election: true
I20260926 01:13:10.975929 16576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98418cdfbca94d9ba37eb367788fa8ee" candidate_uuid: "a44170426a7449ada0cd5417ec05f489" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93e20e9f1d2e4ee28d5d0891a70aa96e" is_pre_election: true
I20260926 01:13:10.976100 16576 raft_consensus.cc:2454] T 98418cdfbca94d9ba37eb367788fa8ee P 93e20e9f1d2e4ee28d5d0891a70aa96e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a44170426a7449ada0cd5417ec05f489 in term 0.
I20260926 01:13:10.976099 16708 raft_consensus.cc:2454] T 98418cdfbca94d9ba37eb367788fa8ee P 32a292318dbc49b8a386537088d5931e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a44170426a7449ada0cd5417ec05f489 in term 0.
I20260926 01:13:10.976308 16370 leader_election.cc:302] T 98418cdfbca94d9ba37eb367788fa8ee P a44170426a7449ada0cd5417ec05f489 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93e20e9f1d2e4ee28d5d0891a70aa96e, a44170426a7449ada0cd5417ec05f489; no voters:
I20260926 01:13:10.976459 16490 raft_consensus.cc:2781] T 98418cdfbca94d9ba37eb367788fa8ee P a44170426a7449ada0cd5417ec05f489 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:13:10.976509 16490 raft_consensus.cc:482] T 98418cdfbca94d9ba37eb367788fa8ee P a44170426a7449ada0cd5417ec05f489 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:13:10.976532 16490 raft_consensus.cc:3037] T 98418cdfbca94d9ba37eb367788fa8ee P a44170426a7449ada0cd5417ec05f489 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:13:10.977288 16490 raft_consensus.cc:504] T 98418cdfbca94d9ba37eb367788fa8ee P a44170426a7449ada0cd5417ec05f489 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e20e9f1d2e4ee28d5d0891a70aa96e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39747 } } peers { permanent_uuid: "32a292318dbc49b8a386537088d5931e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42549 } } peers { permanent_uuid: "a44170426a7449ada0cd5417ec05f489" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41097 } }
I20260926 01:13:10.977449 16490 leader_election.cc:288] T 98418cdfbca94d9ba37eb367788fa8ee P a44170426a7449ada0cd5417ec05f489 [CANDIDATE]: Term 1 election: Requested vote from peers 93e20e9f1d2e4ee28d5d0891a70aa96e (127.0.98.2:39747), 32a292318dbc49b8a386537088d5931e (127.0.98.3:42549)
I20260926 01:13:10.977636 16708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98418cdfbca94d9ba37eb367788fa8ee" candidate_uuid: "a44170426a7449ada0cd5417ec05f489" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32a292318dbc49b8a386537088d5931e"
I20260926 01:13:10.977645 16576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98418cdfbca94d9ba37eb367788fa8ee" candidate_uuid: "a44170426a7449ada0cd5417ec05f489" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93e20e9f1d2e4ee28d5d0891a70aa96e"
I20260926 01:13:10.977720 16708 raft_consensus.cc:3037] T 98418cdfbca94d9ba37eb367788fa8ee P 32a292318dbc49b8a386537088d5931e [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:13:10.977720 16576 raft_consensus.cc:3037] T 98418cdfbca94d9ba37eb367788fa8ee P 93e20e9f1d2e4ee28d5d0891a70aa96e [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:13:10.978466 16708 raft_consensus.cc:2454] T 98418cdfbca94d9ba37eb367788fa8ee P 32a292318dbc49b8a386537088d5931e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a44170426a7449ada0cd5417ec05f489 in term 1.
I20260926 01:13:10.978471 16576 raft_consensus.cc:2454] T 98418cdfbca94d9ba37eb367788fa8ee P 93e20e9f1d2e4ee28d5d0891a70aa96e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a44170426a7449ada0cd5417ec05f489 in term 1.
I20260926 01:13:10.978621 16370 leader_election.cc:302] T 98418cdfbca94d9ba37eb367788fa8ee P a44170426a7449ada0cd5417ec05f489 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93e20e9f1d2e4ee28d5d0891a70aa96e, a44170426a7449ada0cd5417ec05f489; no voters:
I20260926 01:13:10.978726 16490 raft_consensus.cc:2781] T 98418cdfbca94d9ba37eb367788fa8ee P a44170426a7449ada0cd5417ec05f489 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:13:10.978873 16490 raft_consensus.cc:686] T 98418cdfbca94d9ba37eb367788fa8ee P a44170426a7449ada0cd5417ec05f489 [term 1 LEADER]: Becoming Leader. State: Replica: a44170426a7449ada0cd5417ec05f489, State: Running, Role: LEADER
I20260926 01:13:10.978943 16490 consensus_queue.cc:237] T 98418cdfbca94d9ba37eb367788fa8ee P a44170426a7449ada0cd5417ec05f489 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e20e9f1d2e4ee28d5d0891a70aa96e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39747 } } peers { permanent_uuid: "32a292318dbc49b8a386537088d5931e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42549 } } peers { permanent_uuid: "a44170426a7449ada0cd5417ec05f489" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41097 } }
I20260926 01:13:10.979770 16295 catalog_manager.cc:5495] T 98418cdfbca94d9ba37eb367788fa8ee P a44170426a7449ada0cd5417ec05f489 reported cstate change: term changed from 0 to 1, leader changed from <none> to a44170426a7449ada0cd5417ec05f489 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "a44170426a7449ada0cd5417ec05f489" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e20e9f1d2e4ee28d5d0891a70aa96e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32a292318dbc49b8a386537088d5931e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42549 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a44170426a7449ada0cd5417ec05f489" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41097 } health_report { overall_health: HEALTHY } } }
W20260926 01:13:11.000393 16755 tablet.cc:2369] T 98418cdfbca94d9ba37eb367788fa8ee P 32a292318dbc49b8a386537088d5931e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 01:13:11.016417 16774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:11.013131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52972 (local address 127.0.98.1:41097)
0926 01:13:11.013263 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0926 01:13:11.013265 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:13:11.013273 (+ 8us) server_negotiation.cc:407] Connection header received
0926 01:13:11.013335 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:13:11.013336 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:13:11.013355 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:13:11.013421 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:13:11.013754 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:11.014280 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:11.015443 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:11.015654 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:11.015725 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:11.015912 (+ 187us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:13:11.016038 (+ 126us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:13:11.016141 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 01:13:11.016314 (+ 173us) server_negotiation.cc:299] Negotiation successful
0926 01:13:11.016341 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":87,"threads_started":1}
I20260926 01:13:11.018131 16708 raft_consensus.cc:1261] T 98418cdfbca94d9ba37eb367788fa8ee P 32a292318dbc49b8a386537088d5931e [term 1 FOLLOWER]: Refusing update from remote peer a44170426a7449ada0cd5417ec05f489: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:13:11.018146 16576 raft_consensus.cc:1261] T 98418cdfbca94d9ba37eb367788fa8ee P 93e20e9f1d2e4ee28d5d0891a70aa96e [term 1 FOLLOWER]: Refusing update from remote peer a44170426a7449ada0cd5417ec05f489: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:13:11.018527 16771 consensus_queue.cc:1035] T 98418cdfbca94d9ba37eb367788fa8ee P a44170426a7449ada0cd5417ec05f489 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93e20e9f1d2e4ee28d5d0891a70aa96e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:13:11.018659 16490 consensus_queue.cc:1035] T 98418cdfbca94d9ba37eb367788fa8ee P a44170426a7449ada0cd5417ec05f489 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32a292318dbc49b8a386537088d5931e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:13:11.020956 16775 mvcc.cc:205] Tried to move back new op lower bound from 7333417742407651328 to 7333417742250475520. Current Snapshot: MvccSnapshot[applied={T|T < 7333417742407651328}]
I20260926 01:13:11.021854 16777 mvcc.cc:205] Tried to move back new op lower bound from 7333417742407651328 to 7333417742250475520. Current Snapshot: MvccSnapshot[applied={T|T < 7333417742407651328}]
I20260926 01:13:11.022138 16776 mvcc.cc:205] Tried to move back new op lower bound from 7333417742407651328 to 7333417742250475520. Current Snapshot: MvccSnapshot[applied={T|T < 7333417742407651328 or (T in {7333417742407651328})}]
I20260926 01:13:11.023397 16296 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35600:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:13:11.023490 16296 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35600:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:13:11.024348 16296 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 23f32ab05d694df7a44d5716098d6ba7: Sending DeleteTablet for 3 replicas of tablet 98418cdfbca94d9ba37eb367788fa8ee
I20260926 01:13:11.024847 16418 tablet_service.cc:1514] Processing DeleteTablet for tablet 98418cdfbca94d9ba37eb367788fa8ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:13:11 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44364
I20260926 01:13:11.024844 16688 tablet_service.cc:1514] Processing DeleteTablet for tablet 98418cdfbca94d9ba37eb367788fa8ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:13:11 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39924
I20260926 01:13:11.025081 16556 tablet_service.cc:1514] Processing DeleteTablet for tablet 98418cdfbca94d9ba37eb367788fa8ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:13:11 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58278
I20260926 01:13:11.025264 16784 tablet_replica.cc:330] T 98418cdfbca94d9ba37eb367788fa8ee P 32a292318dbc49b8a386537088d5931e: stopping tablet replica
I20260926 01:13:11.025374 16786 tablet_replica.cc:330] T 98418cdfbca94d9ba37eb367788fa8ee P 93e20e9f1d2e4ee28d5d0891a70aa96e: stopping tablet replica
I20260926 01:13:11.025434 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 16334
I20260926 01:13:11.025451 16784 raft_consensus.cc:2229] T 98418cdfbca94d9ba37eb367788fa8ee P 32a292318dbc49b8a386537088d5931e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:13:11.025552 16786 raft_consensus.cc:2229] T 98418cdfbca94d9ba37eb367788fa8ee P 93e20e9f1d2e4ee28d5d0891a70aa96e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:13:11.025614 16784 raft_consensus.cc:2258] T 98418cdfbca94d9ba37eb367788fa8ee P 32a292318dbc49b8a386537088d5931e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:13:11.025651 16786 raft_consensus.cc:2258] T 98418cdfbca94d9ba37eb367788fa8ee P 93e20e9f1d2e4ee28d5d0891a70aa96e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:13:11.026604 16784 ts_tablet_manager.cc:1905] T 98418cdfbca94d9ba37eb367788fa8ee P 32a292318dbc49b8a386537088d5931e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:13:11.027276 16786 ts_tablet_manager.cc:1905] T 98418cdfbca94d9ba37eb367788fa8ee P 93e20e9f1d2e4ee28d5d0891a70aa96e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:13:11.028682 16784 ts_tablet_manager.cc:1918] T 98418cdfbca94d9ba37eb367788fa8ee P 32a292318dbc49b8a386537088d5931e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:13:11.028787 16784 log.cc:1196] T 98418cdfbca94d9ba37eb367788fa8ee P 32a292318dbc49b8a386537088d5931e: Deleting WAL directory at /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-2/wal/wals/98418cdfbca94d9ba37eb367788fa8ee
I20260926 01:13:11.029026 16786 ts_tablet_manager.cc:1918] T 98418cdfbca94d9ba37eb367788fa8ee P 93e20e9f1d2e4ee28d5d0891a70aa96e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:13:11.029129 16786 log.cc:1196] T 98418cdfbca94d9ba37eb367788fa8ee P 93e20e9f1d2e4ee28d5d0891a70aa96e: Deleting WAL directory at /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-1/wal/wals/98418cdfbca94d9ba37eb367788fa8ee
I20260926 01:13:11.029295 16784 ts_tablet_manager.cc:1939] T 98418cdfbca94d9ba37eb367788fa8ee P 32a292318dbc49b8a386537088d5931e: Deleting consensus metadata
I20260926 01:13:11.029418 16786 ts_tablet_manager.cc:1939] T 98418cdfbca94d9ba37eb367788fa8ee P 93e20e9f1d2e4ee28d5d0891a70aa96e: Deleting consensus metadata
I20260926 01:13:11.029949 16281 catalog_manager.cc:4841] TS 32a292318dbc49b8a386537088d5931e (127.0.98.3:42549): tablet 98418cdfbca94d9ba37eb367788fa8ee (table test-table [id=c47dbfbef52741ef9f4fc7df43073f58]) successfully deleted
I20260926 01:13:11.030061 16280 catalog_manager.cc:4841] TS 93e20e9f1d2e4ee28d5d0891a70aa96e (127.0.98.2:39747): tablet 98418cdfbca94d9ba37eb367788fa8ee (table test-table [id=c47dbfbef52741ef9f4fc7df43073f58]) successfully deleted
W20260926 01:13:11.032671 16281 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41097 (error 108)
I20260926 01:13:11.032981 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 16489
I20260926 01:13:11.033749 16760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:11.033164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41097 (local address 127.0.0.1:52974)
0926 01:13:11.033666 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0926 01:13:11.033710 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":481}
W20260926 01:13:11.033913 16281 catalog_manager.cc:4584] TS a44170426a7449ada0cd5417ec05f489 (127.0.98.1:41097): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 98418cdfbca94d9ba37eb367788fa8ee: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41097: connect: Connection refused (error 111)
I20260926 01:13:11.038174 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 16625
I20260926 01:13:11.044008 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 16263
I20260926 01:13:31.049917 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45309
--webserver_interface=127.0.98.62
--webserver_port=42883
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45309
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:13:31.130903 16788 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:13:31.131201 16788 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:13:31.131248 16788 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:13:31.132763 16788 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 01:13:31.132844 16788 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 01:13:31.132865 16788 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 01:13:31.132882 16788 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:13:31.132898 16788 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:13:31.132915 16788 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:13:31.132933 16788 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:13:31.132962 16788 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:13:31.134781 16788 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45309
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45309
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=42883
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16788
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:13:31.135068 16788 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:13:31.135303 16788 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 01:13:31.140317 16788 server_base.cc:1034] running on GCE node
W20260926 01:13:31.140539 16796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:13:31.140480 16793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:13:31.140480 16794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:13:31.140909 16788 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:13:31.141202 16788 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:13:31.142359 16788 hybrid_clock.cc:648] HybridClock initialized: now 1790385211142345 us; error 27 us; skew 500 ppm
Sep 26 01:13:31 dist-test-slave-7b22 krb5kdc[15716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385211, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:13:31.144909 16788 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 01:13:31.145352 16788 webserver.cc:460] Webserver started at http://127.0.98.62:42883/ using document root <none> and password file <none>
I20260926 01:13:31.145565 16788 fs_manager.cc:359] Metadata directory not provided
I20260926 01:13:31.145615 16788 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:13:31.146991 16788 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:13:31.147605 16803 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:13:31.147749 16788 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 01:13:31.147830 16788 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/master-0/wal
uuid: "23f32ab05d694df7a44d5716098d6ba7"
format_stamp: "Formatted at 2026-09-26 01:12:49 on dist-test-slave-7b22"
server_key: "ac8ca28dfa705c2c180098f738f866d9"
server_key_iv: "3deeff1277e5b154c414d19b3d6516b1"
server_key_version: "encryptionkey@0"
I20260926 01:13:31.148198 16788 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:13:31.171614 16788 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:13:31.173533 16788 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:13:31.173668 16788 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:13:31.177446 16788 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45309
I20260926 01:13:31.177492 16855 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45309 every 8 connection(s)
I20260926 01:13:31.177883 16788 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
I20260926 01:13:31.179440 16856 sys_catalog.cc:263] Verifying existing consensus state
I20260926 01:13:31.180145 16856 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23f32ab05d694df7a44d5716098d6ba7: Bootstrap starting.
I20260926 01:13:31.185241 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 16788
I20260926 01:13:31.185524 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:41097
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:45309
--webserver_port=43205
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 01:13:31.188467 16856 log.cc:824] T 00000000000000000000000000000000 P 23f32ab05d694df7a44d5716098d6ba7: Log is configured to *not* fsync() on all Append() calls
I20260926 01:13:31.190706 16856 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23f32ab05d694df7a44d5716098d6ba7: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 01:13:31.190975 16856 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23f32ab05d694df7a44d5716098d6ba7: Bootstrap complete.
I20260926 01:13:31.193024 16856 raft_consensus.cc:348] T 00000000000000000000000000000000 P 23f32ab05d694df7a44d5716098d6ba7 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23f32ab05d694df7a44d5716098d6ba7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45309 } }
I20260926 01:13:31.193254 16856 raft_consensus.cc:729] T 00000000000000000000000000000000 P 23f32ab05d694df7a44d5716098d6ba7 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23f32ab05d694df7a44d5716098d6ba7, State: Initialized, Role: FOLLOWER
I20260926 01:13:31.193403 16856 consensus_queue.cc:260] T 00000000000000000000000000000000 P 23f32ab05d694df7a44d5716098d6ba7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23f32ab05d694df7a44d5716098d6ba7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45309 } }
I20260926 01:13:31.193477 16856 raft_consensus.cc:388] T 00000000000000000000000000000000 P 23f32ab05d694df7a44d5716098d6ba7 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:13:31.193527 16856 raft_consensus.cc:482] T 00000000000000000000000000000000 P 23f32ab05d694df7a44d5716098d6ba7 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:13:31.193573 16856 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 23f32ab05d694df7a44d5716098d6ba7 [term 2 FOLLOWER]: Advancing to term 3
I20260926 01:13:31.194607 16856 raft_consensus.cc:504] T 00000000000000000000000000000000 P 23f32ab05d694df7a44d5716098d6ba7 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23f32ab05d694df7a44d5716098d6ba7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45309 } }
I20260926 01:13:31.194705 16856 leader_election.cc:302] T 00000000000000000000000000000000 P 23f32ab05d694df7a44d5716098d6ba7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 23f32ab05d694df7a44d5716098d6ba7; no voters:
I20260926 01:13:31.194876 16856 leader_election.cc:288] T 00000000000000000000000000000000 P 23f32ab05d694df7a44d5716098d6ba7 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260926 01:13:31.194931 16860 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 23f32ab05d694df7a44d5716098d6ba7 [term 3 FOLLOWER]: Leader election won for term 3
I20260926 01:13:31.195075 16860 raft_consensus.cc:686] T 00000000000000000000000000000000 P 23f32ab05d694df7a44d5716098d6ba7 [term 3 LEADER]: Becoming Leader. State: Replica: 23f32ab05d694df7a44d5716098d6ba7, State: Running, Role: LEADER
I20260926 01:13:31.195151 16856 sys_catalog.cc:564] T 00000000000000000000000000000000 P 23f32ab05d694df7a44d5716098d6ba7 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:13:31.195194 16860 consensus_queue.cc:237] T 00000000000000000000000000000000 P 23f32ab05d694df7a44d5716098d6ba7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23f32ab05d694df7a44d5716098d6ba7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45309 } }
I20260926 01:13:31.195561 16862 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23f32ab05d694df7a44d5716098d6ba7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 23f32ab05d694df7a44d5716098d6ba7. Latest consensus state: current_term: 3 leader_uuid: "23f32ab05d694df7a44d5716098d6ba7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23f32ab05d694df7a44d5716098d6ba7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45309 } } }
I20260926 01:13:31.195569 16861 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23f32ab05d694df7a44d5716098d6ba7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "23f32ab05d694df7a44d5716098d6ba7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23f32ab05d694df7a44d5716098d6ba7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45309 } } }
I20260926 01:13:31.195655 16862 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23f32ab05d694df7a44d5716098d6ba7 [sys.catalog]: This master's current role is: LEADER
I20260926 01:13:31.195672 16861 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23f32ab05d694df7a44d5716098d6ba7 [sys.catalog]: This master's current role is: LEADER
I20260926 01:13:31.195849 16868 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:13:31.196732 16868 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:13:31.196887 16868 catalog_manager.cc:1259] Loaded cluster ID: 49a858af3ce94570a7800a2585b79192
I20260926 01:13:31.197146 16868 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 01:13:31.198347 16868 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:13:31.198591 16868 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 23f32ab05d694df7a44d5716098d6ba7: Loaded TSK: 0
I20260926 01:13:31.198690 16868 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 23f32ab05d694df7a44d5716098d6ba7: Loaded TSK: 1
F20260926 01:13:31.198997 16868 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f8c35e909ad google::LogMessage::Fail() at ??:0
@ 0x7f8c35e9388a google::LogMessage::SendToLog() at ??:0
@ 0x7f8c35e904de google::LogMessage::Flush() at ??:0
@ 0x7f8c35e91a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f8c385fd104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f8c385fad41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f8c3ddea1cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f8c3ddea48e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f8c3ddea594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f8c3ddb0c48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f8c3ddbaad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f8c37d5e0d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f8c37d53954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f8c38bb76db start_thread at ??:0
@ 0x7f8c3498e61f clone at ??:0
W20260926 01:13:31.285115 16858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:13:31.285336 16858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:13:31.285360 16858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:13:31.286680 16858 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:13:31.286731 16858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:13:31.286761 16858 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 01:13:31.288322 16858 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:41097
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=43205
--tserver_master_addrs=127.0.98.62:45309
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16858
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:13:31.288565 16858 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:13:31.288794 16858 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:13:31.293783 16900 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:13:31.293772 16897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:13:31.294006 16858 server_base.cc:1034] running on GCE node
W20260926 01:13:31.293769 16898 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:13:31.294302 16858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:13:31.294548 16858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:13:31.295718 16858 hybrid_clock.cc:648] HybridClock initialized: now 1790385211295695 us; error 33 us; skew 500 ppm
Sep 26 01:13:31 dist-test-slave-7b22 krb5kdc[15716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385211, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:13:31.298529 16858 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 01:13:31.299073 16858 webserver.cc:460] Webserver started at http://127.0.98.1:43205/ using document root <none> and password file <none>
I20260926 01:13:31.299347 16858 fs_manager.cc:359] Metadata directory not provided
I20260926 01:13:31.299408 16858 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:13:31.301182 16858 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 01:13:31.301952 16907 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:13:31.302119 16858 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 01:13:31.302198 16858 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-0/wal
uuid: "a44170426a7449ada0cd5417ec05f489"
format_stamp: "Formatted at 2026-09-26 01:12:49 on dist-test-slave-7b22"
server_key: "df9ded4ee0a9fbc082989a9db96fa076"
server_key_iv: "79f3b89faba1690855051b5d290bf1c4"
server_key_version: "encryptionkey@0"
I20260926 01:13:31.302577 16858 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:13:31.311570 16858 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:13:31.313617 16858 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:13:31.313758 16858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:13:31.314013 16858 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:13:31.314427 16914 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 01:13:31.316316 16858 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 01:13:31.316362 16858 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260926 01:13:31.316396 16858 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 01:13:31.317009 16858 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 01:13:31.317042 16858 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260926 01:13:31.317128 16914 tablet_bootstrap.cc:492] T 98418cdfbca94d9ba37eb367788fa8ee P a44170426a7449ada0cd5417ec05f489: Bootstrap starting.
I20260926 01:13:31.323642 16858 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41097
I20260926 01:13:31.323717 17021 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41097 every 8 connection(s)
I20260926 01:13:31.324088 16858 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-0/data/info.pb
I20260926 01:13:31.328195 16914 log.cc:824] T 98418cdfbca94d9ba37eb367788fa8ee P a44170426a7449ada0cd5417ec05f489: Log is configured to *not* fsync() on all Append() calls
I20260926 01:13:31.330719 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 16858
I20260926 01:13:31.330969 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:39747
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:45309
--webserver_port=39883
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 01:13:31.331008 16914 tablet_bootstrap.cc:492] T 98418cdfbca94d9ba37eb367788fa8ee P a44170426a7449ada0cd5417ec05f489: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 01:13:31.331274 16914 tablet_bootstrap.cc:492] T 98418cdfbca94d9ba37eb367788fa8ee P a44170426a7449ada0cd5417ec05f489: Bootstrap complete.
I20260926 01:13:31.332330 16914 ts_tablet_manager.cc:1397] T 98418cdfbca94d9ba37eb367788fa8ee P a44170426a7449ada0cd5417ec05f489: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.012s
I20260926 01:13:31.333732 16914 raft_consensus.cc:348] T 98418cdfbca94d9ba37eb367788fa8ee P a44170426a7449ada0cd5417ec05f489 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e20e9f1d2e4ee28d5d0891a70aa96e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39747 } } peers { permanent_uuid: "32a292318dbc49b8a386537088d5931e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42549 } } peers { permanent_uuid: "a44170426a7449ada0cd5417ec05f489" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41097 } }
I20260926 01:13:31.334088 16914 raft_consensus.cc:729] T 98418cdfbca94d9ba37eb367788fa8ee P a44170426a7449ada0cd5417ec05f489 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a44170426a7449ada0cd5417ec05f489, State: Initialized, Role: FOLLOWER
I20260926 01:13:31.334270 16914 consensus_queue.cc:260] T 98418cdfbca94d9ba37eb367788fa8ee P a44170426a7449ada0cd5417ec05f489 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e20e9f1d2e4ee28d5d0891a70aa96e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39747 } } peers { permanent_uuid: "32a292318dbc49b8a386537088d5931e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42549 } } peers { permanent_uuid: "a44170426a7449ada0cd5417ec05f489" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41097 } }
I20260926 01:13:31.334568 16914 ts_tablet_manager.cc:1428] T 98418cdfbca94d9ba37eb367788fa8ee P a44170426a7449ada0cd5417ec05f489: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20260926 01:13:31.390021 17024 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:13:31.324703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45309 (local address 127.0.98.1:59471)
0926 01:13:31.328459 (+ 3756us) negotiation.cc:107] Waiting for socket to connect
0926 01:13:31.328516 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0926 01:13:31.329080 (+ 564us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:13:31.389758 (+ 60678us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45309: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":3699,"thread_start_us":74,"threads_started":1}
W20260926 01:13:31.390295 17022 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45309 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:45309: Client connection negotiation failed: client connection to 127.0.98.62:45309: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 01:13:31.390671 17024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:31.390557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45309 (local address 127.0.98.1:55095)
0926 01:13:31.390607 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0926 01:13:31.390648 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260926 01:13:31.391036 17024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:31.390935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45309 (local address 127.0.98.1:59189)
0926 01:13:31.390983 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0926 01:13:31.391010 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260926 01:13:31.391152 17022 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 01:13:31.414111 17026 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:13:31.414327 17026 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:13:31.414351 17026 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:13:31.415524 17026 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:13:31.415565 17026 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:13:31.415591 17026 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 01:13:31.417032 17026 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:39747
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=39883
--tserver_master_addrs=127.0.98.62:45309
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17026
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:13:31.417225 17026 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:13:31.417403 17026 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:13:31.422166 17032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:13:31.422183 17033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:13:31.422170 17035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:13:31.422480 17026 server_base.cc:1034] running on GCE node
I20260926 01:13:31.422676 17026 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:13:31.422873 17026 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:13:31.424012 17026 hybrid_clock.cc:648] HybridClock initialized: now 1790385211423998 us; error 34 us; skew 500 ppm
Sep 26 01:13:31 dist-test-slave-7b22 krb5kdc[15716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385211, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:13:31.426553 17026 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 01:13:31.426972 17026 webserver.cc:460] Webserver started at http://127.0.98.2:39883/ using document root <none> and password file <none>
I20260926 01:13:31.427160 17026 fs_manager.cc:359] Metadata directory not provided
I20260926 01:13:31.427197 17026 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:13:31.428445 17026 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 01:13:31.429075 17042 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:13:31.429221 17026 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 01:13:31.429302 17026 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-1/wal
uuid: "93e20e9f1d2e4ee28d5d0891a70aa96e"
format_stamp: "Formatted at 2026-09-26 01:12:49 on dist-test-slave-7b22"
server_key: "792d916d3335ac55de814f8a8c732b05"
server_key_iv: "a3b5ba14f34ab0797dc72053300ca2e5"
server_key_version: "encryptionkey@0"
I20260926 01:13:31.429656 17026 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:13:31.451184 17026 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:13:31.453207 17026 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:13:31.453347 17026 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:13:31.453574 17026 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:13:31.453935 17026 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:13:31.453974 17026 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:13:31.454020 17026 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:13:31.454044 17026 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:13:31.459765 17026 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39747
I20260926 01:13:31.459820 17155 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39747 every 8 connection(s)
I20260926 01:13:31.460194 17026 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-1/data/info.pb
I20260926 01:13:31.461280 17158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:31.460887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45309 (local address 127.0.98.2:47583)
0926 01:13:31.461069 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0926 01:13:31.461142 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":83,"threads_started":1}
W20260926 01:13:31.461654 17156 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45309 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:45309: Client connection negotiation failed: client connection to 127.0.98.62:45309: connect: Connection refused (error 111)
I20260926 01:13:31.461946 17158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:31.461872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45309 (local address 127.0.98.2:41287)
0926 01:13:31.461902 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0926 01:13:31.461928 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
I20260926 01:13:31.462144 17158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:31.462096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45309 (local address 127.0.98.2:52135)
0926 01:13:31.462115 (+ 19us) negotiation.cc:107] Waiting for socket to connect
0926 01:13:31.462133 (+ 18us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":11}
W20260926 01:13:31.462218 17156 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 01:13:31.467254 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 17026
I20260926 01:13:31.467535 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:42549
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:45309
--webserver_port=43609
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:13:31.547096 17159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:13:31.547312 17159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:13:31.547333 17159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:13:31.548511 17159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:13:31.548553 17159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:13:31.548579 17159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 01:13:31.550097 17159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:42549
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=43609
--tserver_master_addrs=127.0.98.62:45309
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17159
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:13:31.550341 17159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:13:31.550554 17159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:13:31.555326 17165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:13:31.555336 17164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:13:31.555500 17159 server_base.cc:1034] running on GCE node
W20260926 01:13:31.555354 17167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:13:31.555802 17159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:13:31.556037 17159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:13:31.557192 17159 hybrid_clock.cc:648] HybridClock initialized: now 1790385211557180 us; error 30 us; skew 500 ppm
Sep 26 01:13:31 dist-test-slave-7b22 krb5kdc[15716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385211, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:13:31.559737 17159 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 01:13:31.560153 17159 webserver.cc:460] Webserver started at http://127.0.98.3:43609/ using document root <none> and password file <none>
I20260926 01:13:31.560351 17159 fs_manager.cc:359] Metadata directory not provided
I20260926 01:13:31.560401 17159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:13:31.561705 17159 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:13:31.562304 17174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:13:31.562508 17159 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 01:13:31.562580 17159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-2/wal
uuid: "32a292318dbc49b8a386537088d5931e"
format_stamp: "Formatted at 2026-09-26 01:12:49 on dist-test-slave-7b22"
server_key: "40190f19815e04d52f09f97fef649f8e"
server_key_iv: "f9942246e0fa6846e7aeeb7ee71e1f3e"
server_key_version: "encryptionkey@0"
I20260926 01:13:31.562927 17159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:13:31.584616 17159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:13:31.586624 17159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:13:31.586753 17159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:13:31.586984 17159 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:13:31.587292 17159 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:13:31.587329 17159 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:13:31.587352 17159 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:13:31.587364 17159 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:13:31.592908 17159 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42549
I20260926 01:13:31.592968 17287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42549 every 8 connection(s)
I20260926 01:13:31.593286 17159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384993574762-392-0/minicluster-data/ts-2/data/info.pb
I20260926 01:13:31.594340 17290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:31.593959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45309 (local address 127.0.98.3:49007)
0926 01:13:31.594106 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 01:13:31.594179 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":72,"threads_started":1}
W20260926 01:13:31.594790 17288 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45309 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:45309: Client connection negotiation failed: client connection to 127.0.98.62:45309: connect: Connection refused (error 111)
I20260926 01:13:31.595224 17290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:31.595100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45309 (local address 127.0.98.3:53747)
0926 01:13:31.595167 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0926 01:13:31.595201 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260926 01:13:31.595541 17290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:31.595474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45309 (local address 127.0.98.3:53483)
0926 01:13:31.595507 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0926 01:13:31.595527 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260926 01:13:31.595616 17288 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 01:13:31.602638 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 17159
I20260926 01:13:32.391748 17292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:32.391514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45309 (local address 127.0.98.1:36133)
0926 01:13:32.391646 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0926 01:13:32.391684 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":62,"threads_started":1}
I20260926 01:13:32.462775 17293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:32.462535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45309 (local address 127.0.98.2:58197)
0926 01:13:32.462673 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0926 01:13:32.462724 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":57,"threads_started":1}
I20260926 01:13:32.596287 17294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:32.596068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45309 (local address 127.0.98.3:43801)
0926 01:13:32.596187 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0926 01:13:32.596228 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":59,"threads_started":1}
I20260926 01:13:32.713392 17295 raft_consensus.cc:482] T 98418cdfbca94d9ba37eb367788fa8ee P a44170426a7449ada0cd5417ec05f489 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:13:32.713533 17295 raft_consensus.cc:504] T 98418cdfbca94d9ba37eb367788fa8ee P a44170426a7449ada0cd5417ec05f489 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e20e9f1d2e4ee28d5d0891a70aa96e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39747 } } peers { permanent_uuid: "32a292318dbc49b8a386537088d5931e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42549 } } peers { permanent_uuid: "a44170426a7449ada0cd5417ec05f489" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41097 } }
I20260926 01:13:32.713917 17295 leader_election.cc:288] T 98418cdfbca94d9ba37eb367788fa8ee P a44170426a7449ada0cd5417ec05f489 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 93e20e9f1d2e4ee28d5d0891a70aa96e (127.0.98.2:39747), 32a292318dbc49b8a386537088d5931e (127.0.98.3:42549)
Sep 26 01:13:32 dist-test-slave-7b22 krb5kdc[15716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385211, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
Sep 26 01:13:32 dist-test-slave-7b22 krb5kdc[15716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385211, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
I20260926 01:13:32.728430 17292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:32.714253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42549 (local address 127.0.98.1:55555)
0926 01:13:32.714356 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0926 01:13:32.714372 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 01:13:32.714475 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:13:32.715581 (+ 1106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:13:32.715585 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:13:32.716065 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:13:32.716505 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:13:32.716515 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:13:32.717859 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:13:32.717861 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:13:32.718816 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:13:32.718819 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:13:32.719604 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:13:32.719606 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:13:32.719838 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:32.720553 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:13:32.720563 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:13:32.723177 (+ 2614us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:13:32.725459 (+ 2282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:13:32.725461 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:13:32.725461 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:13:32.725895 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:13:32.726533 (+ 638us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:13:32.726534 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:13:32.726535 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:13:32.726725 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:13:32.727676 (+ 951us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:13:32.727687 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:13:32.727923 (+ 236us) client_negotiation.cc:769] Sending connection context
0926 01:13:32.728066 (+ 143us) client_negotiation.cc:240] Negotiation successful
0926 01:13:32.728300 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62,"mutex_wait_us":202}
I20260926 01:13:32.728561 17297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:32.714381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42001 (local address 127.0.98.2:39747)
0926 01:13:32.714590 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0926 01:13:32.714602 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 01:13:32.714782 (+ 180us) server_negotiation.cc:407] Connection header received
0926 01:13:32.715300 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:13:32.715304 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:13:32.715363 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:13:32.715535 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:13:32.716691 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:32.717669 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:32.718959 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:32.719420 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:32.719523 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:32.722988 (+ 3465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:13:32.722993 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:13:32.722993 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:13:32.723006 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:13:32.724967 (+ 1961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:13:32.726051 (+ 1084us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:13:32.726057 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:13:32.726058 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:13:32.726419 (+ 361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:13:32.726890 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:13:32.726892 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:13:32.726892 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:13:32.727544 (+ 652us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:13:32.727669 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0926 01:13:32.728253 (+ 584us) server_negotiation.cc:299] Negotiation successful
0926 01:13:32.728435 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":91,"threads_started":1}
I20260926 01:13:32.728638 17296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:32.714342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55555 (local address 127.0.98.3:42549)
0926 01:13:32.714570 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0926 01:13:32.714589 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 01:13:32.714645 (+ 56us) server_negotiation.cc:407] Connection header received
0926 01:13:32.715253 (+ 608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:13:32.715256 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:13:32.715319 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:13:32.715507 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:13:32.716677 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:32.717701 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:32.718963 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:32.719504 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:32.719632 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:32.723322 (+ 3690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:13:32.723327 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:13:32.723327 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:13:32.723347 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:13:32.725337 (+ 1990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:13:32.726051 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:13:32.726057 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:13:32.726058 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:13:32.726419 (+ 361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:13:32.726890 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:13:32.726892 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:13:32.726892 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:13:32.727543 (+ 651us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:13:32.727669 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0926 01:13:32.728253 (+ 584us) server_negotiation.cc:299] Negotiation successful
0926 01:13:32.728530 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":78,"threads_started":1}
I20260926 01:13:32.728545 17298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:32.714280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39747 (local address 127.0.98.1:42001)
0926 01:13:32.714703 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0926 01:13:32.714713 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 01:13:32.714772 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:13:32.715594 (+ 822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:13:32.715596 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:13:32.716073 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:13:32.716508 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:13:32.716517 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:13:32.717798 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:13:32.717800 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:13:32.718819 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:13:32.718824 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:13:32.719540 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:13:32.719541 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:13:32.719684 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:32.720550 (+ 866us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:13:32.720562 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:13:32.722810 (+ 2248us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:13:32.725158 (+ 2348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:13:32.725175 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:13:32.725185 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:13:32.725895 (+ 710us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:13:32.726553 (+ 658us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:13:32.726554 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:13:32.726554 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:13:32.726729 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:13:32.727693 (+ 964us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:13:32.727694 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:13:32.727923 (+ 229us) client_negotiation.cc:769] Sending connection context
0926 01:13:32.728066 (+ 143us) client_negotiation.cc:240] Negotiation successful
0926 01:13:32.728368 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":405,"mutex_wait_us":525,"thread_start_us":68,"threads_started":1}
W20260926 01:13:32.730324 16909 leader_election.cc:341] T 98418cdfbca94d9ba37eb367788fa8ee P a44170426a7449ada0cd5417ec05f489 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 32a292318dbc49b8a386537088d5931e (127.0.98.3:42549): Not found: Tablet not found: 98418cdfbca94d9ba37eb367788fa8ee
W20260926 01:13:32.730472 16908 leader_election.cc:341] T 98418cdfbca94d9ba37eb367788fa8ee P a44170426a7449ada0cd5417ec05f489 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 93e20e9f1d2e4ee28d5d0891a70aa96e (127.0.98.2:39747): Not found: Tablet not found: 98418cdfbca94d9ba37eb367788fa8ee
I20260926 01:13:32.730517 16908 leader_election.cc:302] T 98418cdfbca94d9ba37eb367788fa8ee P a44170426a7449ada0cd5417ec05f489 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a44170426a7449ada0cd5417ec05f489; no voters: 32a292318dbc49b8a386537088d5931e, 93e20e9f1d2e4ee28d5d0891a70aa96e
I20260926 01:13:32.730648 17295 raft_consensus.cc:2726] T 98418cdfbca94d9ba37eb367788fa8ee P a44170426a7449ada0cd5417ec05f489 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 01:13:33.392587 17299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:33.392322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45309 (local address 127.0.98.1:47513)
0926 01:13:33.392508 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0926 01:13:33.392540 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":75,"threads_started":1}
I20260926 01:13:33.463474 17300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:33.463240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45309 (local address 127.0.98.2:48863)
0926 01:13:33.463381 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0926 01:13:33.463431 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":62,"threads_started":1}
I20260926 01:13:33.596910 17301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:33.596703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45309 (local address 127.0.98.3:49969)
0926 01:13:33.596829 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0926 01:13:33.596869 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":60,"threads_started":1}
I20260926 01:13:34.393455 17302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:34.393169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45309 (local address 127.0.98.1:42949)
0926 01:13:34.393329 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 01:13:34.393381 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":67,"threads_started":1}
I20260926 01:13:34.464073 17303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:34.463883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45309 (local address 127.0.98.2:35389)
0926 01:13:34.463990 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0926 01:13:34.464022 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":69,"threads_started":1}
I20260926 01:13:34.503418 17304 raft_consensus.cc:482] T 98418cdfbca94d9ba37eb367788fa8ee P a44170426a7449ada0cd5417ec05f489 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:13:34.503523 17304 raft_consensus.cc:504] T 98418cdfbca94d9ba37eb367788fa8ee P a44170426a7449ada0cd5417ec05f489 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e20e9f1d2e4ee28d5d0891a70aa96e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39747 } } peers { permanent_uuid: "32a292318dbc49b8a386537088d5931e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42549 } } peers { permanent_uuid: "a44170426a7449ada0cd5417ec05f489" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41097 } }
I20260926 01:13:34.503701 17304 leader_election.cc:288] T 98418cdfbca94d9ba37eb367788fa8ee P a44170426a7449ada0cd5417ec05f489 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 93e20e9f1d2e4ee28d5d0891a70aa96e (127.0.98.2:39747), 32a292318dbc49b8a386537088d5931e (127.0.98.3:42549)
W20260926 01:13:34.504129 16909 leader_election.cc:341] T 98418cdfbca94d9ba37eb367788fa8ee P a44170426a7449ada0cd5417ec05f489 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 32a292318dbc49b8a386537088d5931e (127.0.98.3:42549): Not found: Tablet not found: 98418cdfbca94d9ba37eb367788fa8ee
W20260926 01:13:34.504267 16908 leader_election.cc:341] T 98418cdfbca94d9ba37eb367788fa8ee P a44170426a7449ada0cd5417ec05f489 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 93e20e9f1d2e4ee28d5d0891a70aa96e (127.0.98.2:39747): Not found: Tablet not found: 98418cdfbca94d9ba37eb367788fa8ee
I20260926 01:13:34.504335 16908 leader_election.cc:302] T 98418cdfbca94d9ba37eb367788fa8ee P a44170426a7449ada0cd5417ec05f489 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a44170426a7449ada0cd5417ec05f489; no voters: 32a292318dbc49b8a386537088d5931e, 93e20e9f1d2e4ee28d5d0891a70aa96e
I20260926 01:13:34.504405 17304 raft_consensus.cc:2726] T 98418cdfbca94d9ba37eb367788fa8ee P a44170426a7449ada0cd5417ec05f489 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 01:13:34.597627 17305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:34.597388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45309 (local address 127.0.98.3:55423)
0926 01:13:34.597538 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 01:13:34.597583 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":57,"threads_started":1}
I20260926 01:13:35.394318 17306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:35.394086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45309 (local address 127.0.98.1:42173)
0926 01:13:35.394230 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0926 01:13:35.394272 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":73,"threads_started":1}
I20260926 01:13:35.464931 17307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:35.464650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45309 (local address 127.0.98.2:60739)
0926 01:13:35.464837 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0926 01:13:35.464883 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":89,"threads_started":1}
I20260926 01:13:35.598330 17308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:35.598098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45309 (local address 127.0.98.3:57991)
0926 01:13:35.598244 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0926 01:13:35.598282 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":60,"threads_started":1}
I20260926 01:13:36.395112 17309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:36.394859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45309 (local address 127.0.98.1:37253)
0926 01:13:36.395029 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0926 01:13:36.395067 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":71,"threads_started":1}
I20260926 01:13:36.465729 17310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:36.465458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45309 (local address 127.0.98.2:34663)
0926 01:13:36.465608 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 01:13:36.465680 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":91,"threads_started":1}
I20260926 01:13:36.599141 17311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:36.598879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45309 (local address 127.0.98.3:54361)
0926 01:13:36.599038 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 01:13:36.599093 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":71,"threads_started":1}
I20260926 01:13:37.019626 17312 raft_consensus.cc:482] T 98418cdfbca94d9ba37eb367788fa8ee P a44170426a7449ada0cd5417ec05f489 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:13:37.019733 17312 raft_consensus.cc:504] T 98418cdfbca94d9ba37eb367788fa8ee P a44170426a7449ada0cd5417ec05f489 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e20e9f1d2e4ee28d5d0891a70aa96e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39747 } } peers { permanent_uuid: "32a292318dbc49b8a386537088d5931e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42549 } } peers { permanent_uuid: "a44170426a7449ada0cd5417ec05f489" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41097 } }
I20260926 01:13:37.019909 17312 leader_election.cc:288] T 98418cdfbca94d9ba37eb367788fa8ee P a44170426a7449ada0cd5417ec05f489 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 93e20e9f1d2e4ee28d5d0891a70aa96e (127.0.98.2:39747), 32a292318dbc49b8a386537088d5931e (127.0.98.3:42549)
W20260926 01:13:37.020324 16909 leader_election.cc:341] T 98418cdfbca94d9ba37eb367788fa8ee P a44170426a7449ada0cd5417ec05f489 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 32a292318dbc49b8a386537088d5931e (127.0.98.3:42549): Not found: Tablet not found: 98418cdfbca94d9ba37eb367788fa8ee
W20260926 01:13:37.020404 16908 leader_election.cc:341] T 98418cdfbca94d9ba37eb367788fa8ee P a44170426a7449ada0cd5417ec05f489 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 93e20e9f1d2e4ee28d5d0891a70aa96e (127.0.98.2:39747): Not found: Tablet not found: 98418cdfbca94d9ba37eb367788fa8ee
I20260926 01:13:37.020521 16908 leader_election.cc:302] T 98418cdfbca94d9ba37eb367788fa8ee P a44170426a7449ada0cd5417ec05f489 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a44170426a7449ada0cd5417ec05f489; no voters: 32a292318dbc49b8a386537088d5931e, 93e20e9f1d2e4ee28d5d0891a70aa96e
I20260926 01:13:37.020581 17312 raft_consensus.cc:2726] T 98418cdfbca94d9ba37eb367788fa8ee P a44170426a7449ada0cd5417ec05f489 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 01:13:37.395990 17313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:37.395732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45309 (local address 127.0.98.1:58497)
0926 01:13:37.395895 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 01:13:37.395939 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":82,"threads_started":1}
I20260926 01:13:37.466435 17314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:37.466197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45309 (local address 127.0.98.2:47609)
0926 01:13:37.466334 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0926 01:13:37.466371 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":67,"threads_started":1}
I20260926 01:13:37.599907 17315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:37.599651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45309 (local address 127.0.98.3:40069)
0926 01:13:37.599814 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 01:13:37.599859 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":83,"threads_started":1}
I20260926 01:13:38.396807 17316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:38.396556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45309 (local address 127.0.98.1:34003)
0926 01:13:38.396715 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 01:13:38.396760 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":71,"threads_started":1}
I20260926 01:13:38.467195 17317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:38.466980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45309 (local address 127.0.98.2:51799)
0926 01:13:38.467101 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0926 01:13:38.467145 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":63,"threads_started":1}
I20260926 01:13:38.600677 17318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:38.600420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45309 (local address 127.0.98.3:60053)
0926 01:13:38.600585 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0926 01:13:38.600628 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":85,"threads_started":1}
I20260926 01:13:39.318751 17319 raft_consensus.cc:482] T 98418cdfbca94d9ba37eb367788fa8ee P a44170426a7449ada0cd5417ec05f489 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:13:39.318854 17319 raft_consensus.cc:504] T 98418cdfbca94d9ba37eb367788fa8ee P a44170426a7449ada0cd5417ec05f489 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e20e9f1d2e4ee28d5d0891a70aa96e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39747 } } peers { permanent_uuid: "32a292318dbc49b8a386537088d5931e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42549 } } peers { permanent_uuid: "a44170426a7449ada0cd5417ec05f489" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41097 } }
I20260926 01:13:39.319023 17319 leader_election.cc:288] T 98418cdfbca94d9ba37eb367788fa8ee P a44170426a7449ada0cd5417ec05f489 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 93e20e9f1d2e4ee28d5d0891a70aa96e (127.0.98.2:39747), 32a292318dbc49b8a386537088d5931e (127.0.98.3:42549)
W20260926 01:13:39.319485 16908 leader_election.cc:341] T 98418cdfbca94d9ba37eb367788fa8ee P a44170426a7449ada0cd5417ec05f489 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 93e20e9f1d2e4ee28d5d0891a70aa96e (127.0.98.2:39747): Not found: Tablet not found: 98418cdfbca94d9ba37eb367788fa8ee
W20260926 01:13:39.319574 16909 leader_election.cc:341] T 98418cdfbca94d9ba37eb367788fa8ee P a44170426a7449ada0cd5417ec05f489 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 32a292318dbc49b8a386537088d5931e (127.0.98.3:42549): Not found: Tablet not found: 98418cdfbca94d9ba37eb367788fa8ee
I20260926 01:13:39.319609 16909 leader_election.cc:302] T 98418cdfbca94d9ba37eb367788fa8ee P a44170426a7449ada0cd5417ec05f489 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a44170426a7449ada0cd5417ec05f489; no voters: 32a292318dbc49b8a386537088d5931e, 93e20e9f1d2e4ee28d5d0891a70aa96e
I20260926 01:13:39.319693 17319 raft_consensus.cc:2726] T 98418cdfbca94d9ba37eb367788fa8ee P a44170426a7449ada0cd5417ec05f489 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 01:13:39.397511 17320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:39.397301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45309 (local address 127.0.98.1:34847)
0926 01:13:39.397428 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0926 01:13:39.397463 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":61,"threads_started":1}
I20260926 01:13:39.467954 17321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:39.467711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45309 (local address 127.0.98.2:52313)
0926 01:13:39.467865 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 01:13:39.467908 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":78,"threads_started":1}
I20260926 01:13:39.601464 17322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:39.601240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45309 (local address 127.0.98.3:47331)
0926 01:13:39.601357 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0926 01:13:39.601401 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":64,"threads_started":1}
I20260926 01:13:40.398335 17323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:40.398108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45309 (local address 127.0.98.1:55841)
0926 01:13:40.398242 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0926 01:13:40.398287 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":66,"threads_started":1}
I20260926 01:13:40.468747 17324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:40.468468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45309 (local address 127.0.98.2:35825)
0926 01:13:40.468650 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0926 01:13:40.468699 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":85,"threads_started":1}
I20260926 01:13:40.602260 17325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:40.602021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45309 (local address 127.0.98.3:53185)
0926 01:13:40.602160 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0926 01:13:40.602207 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":62,"threads_started":1}
I20260926 01:13:40.837126 17326 raft_consensus.cc:482] T 98418cdfbca94d9ba37eb367788fa8ee P a44170426a7449ada0cd5417ec05f489 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:13:40.837224 17326 raft_consensus.cc:504] T 98418cdfbca94d9ba37eb367788fa8ee P a44170426a7449ada0cd5417ec05f489 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e20e9f1d2e4ee28d5d0891a70aa96e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39747 } } peers { permanent_uuid: "32a292318dbc49b8a386537088d5931e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42549 } } peers { permanent_uuid: "a44170426a7449ada0cd5417ec05f489" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41097 } }
I20260926 01:13:40.837387 17326 leader_election.cc:288] T 98418cdfbca94d9ba37eb367788fa8ee P a44170426a7449ada0cd5417ec05f489 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 93e20e9f1d2e4ee28d5d0891a70aa96e (127.0.98.2:39747), 32a292318dbc49b8a386537088d5931e (127.0.98.3:42549)
W20260926 01:13:40.837756 16908 leader_election.cc:341] T 98418cdfbca94d9ba37eb367788fa8ee P a44170426a7449ada0cd5417ec05f489 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 93e20e9f1d2e4ee28d5d0891a70aa96e (127.0.98.2:39747): Not found: Tablet not found: 98418cdfbca94d9ba37eb367788fa8ee
W20260926 01:13:40.837841 16909 leader_election.cc:341] T 98418cdfbca94d9ba37eb367788fa8ee P a44170426a7449ada0cd5417ec05f489 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 32a292318dbc49b8a386537088d5931e (127.0.98.3:42549): Not found: Tablet not found: 98418cdfbca94d9ba37eb367788fa8ee
I20260926 01:13:40.837920 16909 leader_election.cc:302] T 98418cdfbca94d9ba37eb367788fa8ee P a44170426a7449ada0cd5417ec05f489 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a44170426a7449ada0cd5417ec05f489; no voters: 32a292318dbc49b8a386537088d5931e, 93e20e9f1d2e4ee28d5d0891a70aa96e
I20260926 01:13:40.837999 17326 raft_consensus.cc:2726] T 98418cdfbca94d9ba37eb367788fa8ee P a44170426a7449ada0cd5417ec05f489 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 01:13:41.399129 17327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:41.398866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45309 (local address 127.0.98.1:57813)
0926 01:13:41.399034 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 01:13:41.399080 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":68,"threads_started":1}
I20260926 01:13:41.469537 17328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:41.469283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45309 (local address 127.0.98.2:58149)
0926 01:13:41.469391 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0926 01:13:41.469427 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":61,"threads_started":1}
I20260926 01:13:41.603065 17329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:41.602808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45309 (local address 127.0.98.3:50169)
0926 01:13:41.602951 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 01:13:41.603000 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":65,"threads_started":1}
I20260926 01:13:42.400014 17330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:42.399729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45309 (local address 127.0.98.1:55057)
0926 01:13:42.399900 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0926 01:13:42.399947 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":85,"threads_started":1}
I20260926 01:13:42.470254 17331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:42.469997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45309 (local address 127.0.98.2:60317)
0926 01:13:42.470158 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0926 01:13:42.470199 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":74,"threads_started":1}
I20260926 01:13:42.603771 17332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:42.603532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45309 (local address 127.0.98.3:60585)
0926 01:13:42.603684 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0926 01:13:42.603723 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":78,"threads_started":1}
I20260926 01:13:43.400787 17333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:43.400552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45309 (local address 127.0.98.1:40631)
0926 01:13:43.400693 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0926 01:13:43.400738 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":81,"threads_started":1}
I20260926 01:13:43.471056 17334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:43.470813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45309 (local address 127.0.98.2:45339)
0926 01:13:43.470949 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0926 01:13:43.470996 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":61,"threads_started":1}
I20260926 01:13:43.604492 17335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:43.604219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45309 (local address 127.0.98.3:46689)
0926 01:13:43.604404 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 01:13:43.604445 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":65,"threads_started":1}
I20260926 01:13:44.401595 17336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:44.401336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45309 (local address 127.0.98.1:34781)
0926 01:13:44.401502 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 01:13:44.401545 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":86,"threads_started":1}
I20260926 01:13:44.471807 17337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:44.471584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45309 (local address 127.0.98.2:43261)
0926 01:13:44.471713 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0926 01:13:44.471758 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":68,"threads_started":1}
I20260926 01:13:44.605567 17338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:44.605188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45309 (local address 127.0.98.3:56215)
0926 01:13:44.605465 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0926 01:13:44.605508 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":101,"threads_started":1}
I20260926 01:13:45.402629 17339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:45.402338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45309 (local address 127.0.98.1:54917)
0926 01:13:45.402503 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0926 01:13:45.402579 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":69,"threads_started":1}
I20260926 01:13:45.472678 17340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:45.472433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45309 (local address 127.0.98.2:33861)
0926 01:13:45.472576 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 01:13:45.472626 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":66,"threads_started":1}
I20260926 01:13:45.606511 17341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:45.606221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45309 (local address 127.0.98.3:50327)
0926 01:13:45.606401 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0926 01:13:45.606453 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":85,"threads_started":1}
I20260926 01:13:46.403518 17342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:46.403269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45309 (local address 127.0.98.1:37813)
0926 01:13:46.403421 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0926 01:13:46.403468 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":62,"threads_started":1}
I20260926 01:13:46.473532 17343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:46.473288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45309 (local address 127.0.98.2:46607)
0926 01:13:46.473431 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 01:13:46.473486 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":64,"threads_started":1}
I20260926 01:13:46.607018 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 16858
I20260926 01:13:46.607811 17344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:46.607178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45309 (local address 127.0.98.3:42933)
0926 01:13:46.607726 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0926 01:13:46.607769 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":55,"threads_started":1}
I20260926 01:13:46.611840 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 17026
I20260926 01:13:46.615888 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 17159
2026-09-26T01:13:46Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58784 ms)
[----------] 30 tests from SecurityITest (233044 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384993574762-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 01:13:46 dist-test-slave-7b22 krb5kdc[17347](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 01:13:46 dist-test-slave-7b22 krb5kdc[17347](info): set up 2 sockets
Sep 26 01:13:46 dist-test-slave-7b22 krb5kdc[17347](info): commencing operation
krb5kdc: starting...
W20260926 01:13:47.577322 392 mini_kdc.cc:121] Time spent starting KDC: real 0.942s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384993574762-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384993574762-392-0/krb5kdc/test-user.keytab.
Sep 26 01:13:47 dist-test-slave-7b22 krb5kdc[17347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385227, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T01:13:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:13:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384993574762-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384993574762-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 01:13:47.696369 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384993574762-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384993574762-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384993574762-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35035
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384993574762-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35035
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:13:47.776825 17360 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:13:47.777050 17360 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:13:47.777081 17360 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:13:47.778306 17360 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:13:47.778349 17360 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:13:47.778366 17360 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:13:47.778383 17360 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:13:47.778398 17360 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:13:47.780009 17360 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384993574762-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384993574762-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35035
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384993574762-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35035
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17360
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384993574762-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:13:47.780283 17360 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:13:47.780499 17360 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 01:13:47.785459 17360 server_base.cc:1034] running on GCE node
W20260926 01:13:47.785539 17368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:13:47.785636 17366 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:13:47.785432 17365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:13:47.786026 17360 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:13:47.786341 17360 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:13:47.787496 17360 hybrid_clock.cc:648] HybridClock initialized: now 1790385227787478 us; error 34 us; skew 500 ppm
Sep 26 01:13:47 dist-test-slave-7b22 krb5kdc[17347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385227, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:13:47.789950 17360 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 01:13:47.790392 17360 webserver.cc:460] Webserver started at http://127.0.98.62:37067/ using document root <none> and password file <none>
I20260926 01:13:47.790601 17360 fs_manager.cc:359] Metadata directory not provided
I20260926 01:13:47.790648 17360 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:13:47.790772 17360 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:13:47.791864 17360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384993574762-392-0/minicluster-data/master-0/data/instance:
uuid: "0ed585b0113543ffba64a64ec6dd9c62"
format_stamp: "Formatted at 2026-09-26 01:13:47 on dist-test-slave-7b22"
server_key: "f7d405e5f6ca40e3afba57c4046dabc9"
server_key_iv: "68bd5f5c43d9d5fdbf617dbd1858d197"
server_key_version: "encryptionkey@0"
I20260926 01:13:47.792191 17360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384993574762-392-0/minicluster-data/master-0/wal/instance:
uuid: "0ed585b0113543ffba64a64ec6dd9c62"
format_stamp: "Formatted at 2026-09-26 01:13:47 on dist-test-slave-7b22"
server_key: "f7d405e5f6ca40e3afba57c4046dabc9"
server_key_iv: "68bd5f5c43d9d5fdbf617dbd1858d197"
server_key_version: "encryptionkey@0"
I20260926 01:13:47.793426 17360 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 01:13:47.794118 17375 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:13:47.794266 17360 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 01:13:47.794348 17360 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384993574762-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384993574762-392-0/minicluster-data/master-0/wal
uuid: "0ed585b0113543ffba64a64ec6dd9c62"
format_stamp: "Formatted at 2026-09-26 01:13:47 on dist-test-slave-7b22"
server_key: "f7d405e5f6ca40e3afba57c4046dabc9"
server_key_iv: "68bd5f5c43d9d5fdbf617dbd1858d197"
server_key_version: "encryptionkey@0"
I20260926 01:13:47.794409 17360 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384993574762-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384993574762-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384993574762-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:13:47.830505 17360 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:13:47.832544 17360 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:13:47.832681 17360 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:13:47.836582 17360 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35035
I20260926 01:13:47.836939 17360 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
I20260926 01:13:47.836933 17427 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35035 every 8 connection(s)
I20260926 01:13:47.837597 17428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:13:47.839820 17428 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ed585b0113543ffba64a64ec6dd9c62: Bootstrap starting.
I20260926 01:13:47.840359 17428 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0ed585b0113543ffba64a64ec6dd9c62: Neither blocks nor log segments found. Creating new log.
I20260926 01:13:47.840632 17428 log.cc:824] T 00000000000000000000000000000000 P 0ed585b0113543ffba64a64ec6dd9c62: Log is configured to *not* fsync() on all Append() calls
I20260926 01:13:47.841262 17428 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ed585b0113543ffba64a64ec6dd9c62: No bootstrap required, opened a new log
I20260926 01:13:47.841919 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 17360
I20260926 01:13:47.842032 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384993574762-392-0/minicluster-data/master-0/wal/instance
I20260926 01:13:47.842217 392 external_mini_cluster.cc:1428] Setting key ddfe2fcfdce06ac985907dee2e4781e3
I20260926 01:13:47.843226 17428 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0ed585b0113543ffba64a64ec6dd9c62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ed585b0113543ffba64a64ec6dd9c62" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35035 } }
I20260926 01:13:47.843359 17428 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0ed585b0113543ffba64a64ec6dd9c62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:13:47.843391 17428 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0ed585b0113543ffba64a64ec6dd9c62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed585b0113543ffba64a64ec6dd9c62, State: Initialized, Role: FOLLOWER
I20260926 01:13:47.843530 17428 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ed585b0113543ffba64a64ec6dd9c62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ed585b0113543ffba64a64ec6dd9c62" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35035 } }
I20260926 01:13:47.843614 17428 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0ed585b0113543ffba64a64ec6dd9c62 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:13:47.843652 17428 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0ed585b0113543ffba64a64ec6dd9c62 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:13:47.843698 17428 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0ed585b0113543ffba64a64ec6dd9c62 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:13:47.844566 17428 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0ed585b0113543ffba64a64ec6dd9c62 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ed585b0113543ffba64a64ec6dd9c62" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35035 } }
I20260926 01:13:47.844667 17428 leader_election.cc:302] T 00000000000000000000000000000000 P 0ed585b0113543ffba64a64ec6dd9c62 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0ed585b0113543ffba64a64ec6dd9c62; no voters:
I20260926 01:13:47.844818 17428 leader_election.cc:288] T 00000000000000000000000000000000 P 0ed585b0113543ffba64a64ec6dd9c62 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:13:47.844851 17433 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0ed585b0113543ffba64a64ec6dd9c62 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:13:47.844946 17433 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0ed585b0113543ffba64a64ec6dd9c62 [term 1 LEADER]: Becoming Leader. State: Replica: 0ed585b0113543ffba64a64ec6dd9c62, State: Running, Role: LEADER
I20260926 01:13:47.845067 17428 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0ed585b0113543ffba64a64ec6dd9c62 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:13:47.845082 17433 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0ed585b0113543ffba64a64ec6dd9c62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ed585b0113543ffba64a64ec6dd9c62" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35035 } }
I20260926 01:13:47.845434 17433 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ed585b0113543ffba64a64ec6dd9c62 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0ed585b0113543ffba64a64ec6dd9c62. Latest consensus state: current_term: 1 leader_uuid: "0ed585b0113543ffba64a64ec6dd9c62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ed585b0113543ffba64a64ec6dd9c62" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35035 } } }
I20260926 01:13:47.845578 17433 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ed585b0113543ffba64a64ec6dd9c62 [sys.catalog]: This master's current role is: LEADER
I20260926 01:13:47.846011 17441 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:13:47.846527 17441 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:13:47.848079 17441 catalog_manager.cc:1347] Generated new cluster ID: ff66d44de4c74838abca4fabb6305d5e
I20260926 01:13:47.848136 17441 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 01:13:47.847055 17434 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ed585b0113543ffba64a64ec6dd9c62 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0ed585b0113543ffba64a64ec6dd9c62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ed585b0113543ffba64a64ec6dd9c62" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35035 } } }
I20260926 01:13:47.848362 17434 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ed585b0113543ffba64a64ec6dd9c62 [sys.catalog]: This master's current role is: LEADER
Sep 26 01:13:47 dist-test-slave-7b22 krb5kdc[17347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385227, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:13:47.853055 17432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:47.842729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52550 (local address 127.0.98.62:35035)
0926 01:13:47.842917 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 01:13:47.842925 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 01:13:47.842948 (+ 23us) server_negotiation.cc:407] Connection header received
0926 01:13:47.843188 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:13:47.843193 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:13:47.843244 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:13:47.843357 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:13:47.843995 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:47.844987 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:47.845980 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:47.846579 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:47.846682 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:47.849307 (+ 2625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:13:47.849311 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:13:47.849311 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:13:47.849323 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:13:47.851226 (+ 1903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:13:47.851604 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:13:47.851607 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:13:47.851607 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:13:47.851738 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:13:47.851911 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:13:47.851912 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:13:47.851912 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:13:47.852244 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:13:47.852399 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0926 01:13:47.852472 (+ 73us) server_negotiation.cc:299] Negotiation successful
0926 01:13:47.852724 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":73,"threads_started":1}
I20260926 01:13:47.867609 17441 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:13:47.868139 17441 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:13:47.870872 17441 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0ed585b0113543ffba64a64ec6dd9c62: Generated new TSK 0
I20260926 01:13:47.871042 17441 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384993574762-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384993574762-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 01:13:47.901042 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384993574762-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384993574762-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384993574762-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384993574762-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35035
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384993574762-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:13:47.981585 17454 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:13:47.981904 17454 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:13:47.981942 17454 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:13:47.983340 17454 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:13:47.983389 17454 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:13:47.983417 17454 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 01:13:47.984886 17454 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384993574762-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384993574762-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384993574762-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384993574762-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35035
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17454
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384993574762-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:13:47.985077 17454 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:13:47.985280 17454 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 01:13:47.990072 17454 server_base.cc:1034] running on GCE node
W20260926 01:13:47.990043 17462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:13:47.990166 17460 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:13:47.990310 17459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:13:47.990526 17454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:13:47.990793 17454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:13:47.991969 17454 hybrid_clock.cc:648] HybridClock initialized: now 1790385227991949 us; error 33 us; skew 500 ppm
Sep 26 01:13:47 dist-test-slave-7b22 krb5kdc[17347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385227, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:13:47.995028 17454 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 01:13:47.995486 17454 webserver.cc:460] Webserver started at http://127.0.98.1:46799/ using document root <none> and password file <none>
I20260926 01:13:47.995707 17454 fs_manager.cc:359] Metadata directory not provided
I20260926 01:13:47.995761 17454 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:13:47.995893 17454 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:13:47.996959 17454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384993574762-392-0/minicluster-data/ts-0/data/instance:
uuid: "79311ceddd994d968a9a3376aabe3f79"
format_stamp: "Formatted at 2026-09-26 01:13:47 on dist-test-slave-7b22"
server_key: "501ec4e25bf121e48e7dad3c0d065e1b"
server_key_iv: "72eef8ed22ecebefefdf596c57769d77"
server_key_version: "encryptionkey@0"
I20260926 01:13:47.997306 17454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384993574762-392-0/minicluster-data/ts-0/wal/instance:
uuid: "79311ceddd994d968a9a3376aabe3f79"
format_stamp: "Formatted at 2026-09-26 01:13:47 on dist-test-slave-7b22"
server_key: "501ec4e25bf121e48e7dad3c0d065e1b"
server_key_iv: "72eef8ed22ecebefefdf596c57769d77"
server_key_version: "encryptionkey@0"
I20260926 01:13:47.998654 17454 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 01:13:47.999332 17469 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:13:47.999459 17454 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 01:13:47.999536 17454 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384993574762-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384993574762-392-0/minicluster-data/ts-0/wal
uuid: "79311ceddd994d968a9a3376aabe3f79"
format_stamp: "Formatted at 2026-09-26 01:13:47 on dist-test-slave-7b22"
server_key: "501ec4e25bf121e48e7dad3c0d065e1b"
server_key_iv: "72eef8ed22ecebefefdf596c57769d77"
server_key_version: "encryptionkey@0"
I20260926 01:13:47.999603 17454 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384993574762-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384993574762-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384993574762-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:13:48.023839 17454 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:13:48.025810 17454 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:13:48.025951 17454 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:13:48.026207 17454 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:13:48.026508 17454 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:13:48.026542 17454 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:13:48.026571 17454 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:13:48.026587 17454 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:13:48.032037 17454 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35633
I20260926 01:13:48.032080 17582 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35633 every 8 connection(s)
I20260926 01:13:48.032392 17454 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384993574762-392-0/minicluster-data/ts-0/data/info.pb
I20260926 01:13:48.036391 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 17454
I20260926 01:13:48.036499 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384993574762-392-0/minicluster-data/ts-0/wal/instance
I20260926 01:13:48.036688 392 external_mini_cluster.cc:1428] Setting key 7a34eec871db0bcea4578716272c7431
Sep 26 01:13:48 dist-test-slave-7b22 krb5kdc[17347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385227, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:13:48.042754 17432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:48.033189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60817 (local address 127.0.98.62:35035)
0926 01:13:48.033283 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0926 01:13:48.033285 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:13:48.033439 (+ 154us) server_negotiation.cc:407] Connection header received
0926 01:13:48.034502 (+ 1063us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:13:48.034504 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:13:48.034567 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:13:48.034640 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:13:48.035278 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:48.035853 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:48.036928 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:48.037226 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:48.037287 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:48.039707 (+ 2420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:13:48.039708 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:13:48.039709 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:13:48.039719 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:13:48.041088 (+ 1369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:13:48.041640 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:13:48.041641 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:13:48.041642 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:13:48.041702 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:13:48.042033 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:13:48.042034 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:13:48.042035 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:13:48.042167 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:13:48.042230 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 01:13:48.042582 (+ 352us) server_negotiation.cc:299] Negotiation successful
0926 01:13:48.042678 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 01:13:48.042980 17585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:48.033107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35035 (local address 127.0.98.1:60817)
0926 01:13:48.033312 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0926 01:13:48.033356 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 01:13:48.034362 (+ 1006us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:13:48.034700 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:13:48.034704 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:13:48.034988 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:13:48.035191 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:13:48.035194 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:13:48.035944 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:13:48.035946 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:13:48.036850 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:13:48.036853 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:13:48.037305 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:13:48.037306 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:13:48.037436 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:48.038034 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:13:48.038042 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:13:48.039596 (+ 1554us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:13:48.041186 (+ 1590us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:13:48.041197 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:13:48.041200 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:13:48.041522 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:13:48.041775 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:13:48.041776 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:13:48.041776 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:13:48.041951 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:13:48.042237 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:13:48.042243 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:13:48.042428 (+ 185us) client_negotiation.cc:769] Sending connection context
0926 01:13:48.042546 (+ 118us) client_negotiation.cc:240] Negotiation successful
0926 01:13:48.042732 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":80,"threads_started":1}
I20260926 01:13:48.043535 17583 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35035
I20260926 01:13:48.043645 17583 heartbeater.cc:464] Registering TS with master...
I20260926 01:13:48.043853 17583 heartbeater.cc:511] Master 127.0.98.62:35035 requested a full tablet report, sending...
I20260926 01:13:48.044409 17392 ts_manager.cc:194] Registered new tserver with Master: 79311ceddd994d968a9a3376aabe3f79 (127.0.98.1:35633)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260926 01:13:48.045121 17392 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:60817
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384993574762-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384993574762-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 01:13:48.064406 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384993574762-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384993574762-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384993574762-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384993574762-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35035
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384993574762-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:13:48.150813 17588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:13:48.151033 17588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:13:48.151058 17588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:13:48.152323 17588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:13:48.152369 17588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:13:48.152396 17588 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 01:13:48.154206 17588 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384993574762-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384993574762-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384993574762-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384993574762-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35035
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17588
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384993574762-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:13:48.154480 17588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:13:48.154718 17588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:13:48.159667 17593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:13:48.159616 17594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:13:48.159718 17588 server_base.cc:1034] running on GCE node
W20260926 01:13:48.159623 17596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:13:48.160012 17588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:13:48.160269 17588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:13:48.161427 17588 hybrid_clock.cc:648] HybridClock initialized: now 1790385228161411 us; error 29 us; skew 500 ppm
Sep 26 01:13:48 dist-test-slave-7b22 krb5kdc[17347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385228, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:13:48.163959 17588 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 01:13:48.164371 17588 webserver.cc:460] Webserver started at http://127.0.98.2:44003/ using document root <none> and password file <none>
I20260926 01:13:48.164605 17588 fs_manager.cc:359] Metadata directory not provided
I20260926 01:13:48.164669 17588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:13:48.164793 17588 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:13:48.165908 17588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384993574762-392-0/minicluster-data/ts-1/data/instance:
uuid: "7ce6aae72e374c7a92d1e2f7dfdf197b"
format_stamp: "Formatted at 2026-09-26 01:13:48 on dist-test-slave-7b22"
server_key: "00ced64c1dd8405fe3992f9cc81907b6"
server_key_iv: "8a4cf91e1f4c02b2e0710c8508278a24"
server_key_version: "encryptionkey@0"
I20260926 01:13:48.166235 17588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384993574762-392-0/minicluster-data/ts-1/wal/instance:
uuid: "7ce6aae72e374c7a92d1e2f7dfdf197b"
format_stamp: "Formatted at 2026-09-26 01:13:48 on dist-test-slave-7b22"
server_key: "00ced64c1dd8405fe3992f9cc81907b6"
server_key_iv: "8a4cf91e1f4c02b2e0710c8508278a24"
server_key_version: "encryptionkey@0"
I20260926 01:13:48.167541 17588 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 01:13:48.168327 17603 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:13:48.168542 17588 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 01:13:48.168643 17588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384993574762-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384993574762-392-0/minicluster-data/ts-1/wal
uuid: "7ce6aae72e374c7a92d1e2f7dfdf197b"
format_stamp: "Formatted at 2026-09-26 01:13:48 on dist-test-slave-7b22"
server_key: "00ced64c1dd8405fe3992f9cc81907b6"
server_key_iv: "8a4cf91e1f4c02b2e0710c8508278a24"
server_key_version: "encryptionkey@0"
I20260926 01:13:48.168720 17588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384993574762-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384993574762-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384993574762-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:13:48.188522 17588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:13:48.190533 17588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:13:48.190675 17588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:13:48.190882 17588 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:13:48.191190 17588 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:13:48.191226 17588 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:13:48.191257 17588 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:13:48.191282 17588 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:13:48.196666 17588 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43985
I20260926 01:13:48.196713 17716 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43985 every 8 connection(s)
I20260926 01:13:48.197026 17588 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384993574762-392-0/minicluster-data/ts-1/data/info.pb
I20260926 01:13:48.200335 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 17588
I20260926 01:13:48.200454 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384993574762-392-0/minicluster-data/ts-1/wal/instance
I20260926 01:13:48.200645 392 external_mini_cluster.cc:1428] Setting key 2ae4fc6637f26a75c9b305b6e2332d9c
Sep 26 01:13:48 dist-test-slave-7b22 krb5kdc[17347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385228, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:13:48.207589 17432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:48.197729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44619 (local address 127.0.98.62:35035)
0926 01:13:48.197786 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 01:13:48.197788 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:13:48.198018 (+ 230us) server_negotiation.cc:407] Connection header received
0926 01:13:48.198939 (+ 921us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:13:48.198940 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:13:48.198952 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:13:48.199005 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:13:48.199665 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:48.200221 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:48.201604 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:48.201910 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:48.201986 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:48.204421 (+ 2435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:13:48.204422 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:13:48.204423 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:13:48.204431 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:13:48.206073 (+ 1642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:13:48.206458 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:13:48.206460 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:13:48.206460 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:13:48.206525 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:13:48.206867 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:13:48.206868 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:13:48.206868 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:13:48.207016 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:13:48.207088 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 01:13:48.207382 (+ 294us) server_negotiation.cc:299] Negotiation successful
0926 01:13:48.207478 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260926 01:13:48.207669 17719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:48.197705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35035 (local address 127.0.98.2:44619)
0926 01:13:48.197896 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0926 01:13:48.197937 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 01:13:48.198850 (+ 913us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:13:48.199062 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:13:48.199065 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:13:48.199327 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:13:48.199520 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:13:48.199524 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:13:48.200334 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:13:48.200335 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:13:48.201496 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:13:48.201499 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:13:48.201994 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:13:48.201995 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:13:48.202125 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:48.202696 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:13:48.202706 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:13:48.204323 (+ 1617us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:13:48.206154 (+ 1831us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:13:48.206160 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:13:48.206162 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:13:48.206388 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:13:48.206675 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:13:48.206676 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:13:48.206677 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:13:48.206795 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:13:48.207107 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:13:48.207112 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:13:48.207244 (+ 132us) client_negotiation.cc:769] Sending connection context
0926 01:13:48.207341 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 01:13:48.207479 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":78,"threads_started":1}
I20260926 01:13:48.208159 17717 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35035
I20260926 01:13:48.208256 17717 heartbeater.cc:464] Registering TS with master...
I20260926 01:13:48.208429 17717 heartbeater.cc:511] Master 127.0.98.62:35035 requested a full tablet report, sending...
I20260926 01:13:48.208789 17392 ts_manager.cc:194] Registered new tserver with Master: 7ce6aae72e374c7a92d1e2f7dfdf197b (127.0.98.2:43985)
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260926 01:13:48.209203 17392 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:44619
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384993574762-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384993574762-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 01:13:48.229521 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384993574762-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384993574762-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384993574762-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384993574762-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35035
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384993574762-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:13:48.312409 17722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:13:48.312695 17722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:13:48.312724 17722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:13:48.314039 17722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:13:48.314088 17722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:13:48.314118 17722 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 01:13:48.315694 17722 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384993574762-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384993574762-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384993574762-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384993574762-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35035
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17722
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384993574762-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:13:48.315914 17722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:13:48.316110 17722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 01:13:48.321375 17722 server_base.cc:1034] running on GCE node
W20260926 01:13:48.321331 17730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:13:48.321326 17728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:13:48.321324 17727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:13:48.321903 17722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:13:48.322175 17722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:13:48.323343 17722 hybrid_clock.cc:648] HybridClock initialized: now 1790385228323332 us; error 30 us; skew 500 ppm
Sep 26 01:13:48 dist-test-slave-7b22 krb5kdc[17347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385228, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:13:48.325786 17722 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 01:13:48.326275 17722 webserver.cc:460] Webserver started at http://127.0.98.3:35999/ using document root <none> and password file <none>
I20260926 01:13:48.326480 17722 fs_manager.cc:359] Metadata directory not provided
I20260926 01:13:48.326529 17722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:13:48.326651 17722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:13:48.327697 17722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384993574762-392-0/minicluster-data/ts-2/data/instance:
uuid: "15ea5b9f9cad47ddaecff3b98ca4c7d0"
format_stamp: "Formatted at 2026-09-26 01:13:48 on dist-test-slave-7b22"
server_key: "3a0897023116a3911353138cf0993460"
server_key_iv: "b2361567c0a87e32f10eeea9605d66f2"
server_key_version: "encryptionkey@0"
I20260926 01:13:48.328017 17722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384993574762-392-0/minicluster-data/ts-2/wal/instance:
uuid: "15ea5b9f9cad47ddaecff3b98ca4c7d0"
format_stamp: "Formatted at 2026-09-26 01:13:48 on dist-test-slave-7b22"
server_key: "3a0897023116a3911353138cf0993460"
server_key_iv: "b2361567c0a87e32f10eeea9605d66f2"
server_key_version: "encryptionkey@0"
I20260926 01:13:48.329252 17722 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 01:13:48.329943 17737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:13:48.330080 17722 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 01:13:48.330175 17722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384993574762-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384993574762-392-0/minicluster-data/ts-2/wal
uuid: "15ea5b9f9cad47ddaecff3b98ca4c7d0"
format_stamp: "Formatted at 2026-09-26 01:13:48 on dist-test-slave-7b22"
server_key: "3a0897023116a3911353138cf0993460"
server_key_iv: "b2361567c0a87e32f10eeea9605d66f2"
server_key_version: "encryptionkey@0"
I20260926 01:13:48.330242 17722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384993574762-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384993574762-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384993574762-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:13:48.342020 17722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:13:48.343999 17722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:13:48.344134 17722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:13:48.344360 17722 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:13:48.344702 17722 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:13:48.344738 17722 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:13:48.344758 17722 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:13:48.344770 17722 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:13:48.350660 17722 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44299
I20260926 01:13:48.350721 17850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44299 every 8 connection(s)
I20260926 01:13:48.351029 17722 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384993574762-392-0/minicluster-data/ts-2/data/info.pb
I20260926 01:13:48.354665 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 17722
I20260926 01:13:48.354770 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384993574762-392-0/minicluster-data/ts-2/wal/instance
I20260926 01:13:48.354939 392 external_mini_cluster.cc:1428] Setting key 1022bd281b3c89bb397939a6dab31e4a
Sep 26 01:13:48 dist-test-slave-7b22 krb5kdc[17347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385228, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:13:48.360743 17432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:48.351701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34159 (local address 127.0.98.62:35035)
0926 01:13:48.351746 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 01:13:48.351748 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:13:48.351996 (+ 248us) server_negotiation.cc:407] Connection header received
0926 01:13:48.352993 (+ 997us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:13:48.352995 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:13:48.353008 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:13:48.353059 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:13:48.353718 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:48.354312 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:48.355577 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:48.355766 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:48.355820 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:48.358097 (+ 2277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:13:48.358099 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:13:48.358100 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:13:48.358107 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:13:48.359346 (+ 1239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:13:48.359811 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:13:48.359813 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:13:48.359813 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:13:48.359888 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:13:48.360153 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:13:48.360154 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:13:48.360155 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:13:48.360284 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:13:48.360345 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 01:13:48.360603 (+ 258us) server_negotiation.cc:299] Negotiation successful
0926 01:13:48.360674 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 01:13:48.360929 17853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:48.351711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35035 (local address 127.0.98.3:34159)
0926 01:13:48.351877 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 01:13:48.351914 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 01:13:48.352881 (+ 967us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:13:48.353133 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:13:48.353137 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:13:48.353411 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:13:48.353606 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:13:48.353610 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:13:48.354400 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:13:48.354401 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:13:48.355499 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:13:48.355502 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:13:48.355832 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:13:48.355833 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:13:48.355959 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:48.356448 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:13:48.356456 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:13:48.357972 (+ 1516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:13:48.359448 (+ 1476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:13:48.359455 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:13:48.359457 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:13:48.359715 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:13:48.359955 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:13:48.359956 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:13:48.359956 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:13:48.360064 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:13:48.360345 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:13:48.360350 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:13:48.360478 (+ 128us) client_negotiation.cc:769] Sending connection context
0926 01:13:48.360566 (+ 88us) client_negotiation.cc:240] Negotiation successful
0926 01:13:48.360715 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":80,"threads_started":1}
I20260926 01:13:48.361385 17851 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35035
I20260926 01:13:48.361476 17851 heartbeater.cc:464] Registering TS with master...
I20260926 01:13:48.361662 17851 heartbeater.cc:511] Master 127.0.98.62:35035 requested a full tablet report, sending...
I20260926 01:13:48.362108 17392 ts_manager.cc:194] Registered new tserver with Master: 15ea5b9f9cad47ddaecff3b98ca4c7d0 (127.0.98.3:44299)
I20260926 01:13:48.362469 17392 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:34159
I20260926 01:13:48.366608 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 01:13:48.373989 17432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:48.367935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52566 (local address 127.0.98.62:35035)
0926 01:13:48.367991 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 01:13:48.367993 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:13:48.368074 (+ 81us) server_negotiation.cc:407] Connection header received
0926 01:13:48.368121 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:13:48.368121 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:13:48.368131 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:13:48.368181 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:13:48.368706 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:48.369189 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:48.369921 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:48.370120 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:48.370173 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:48.371517 (+ 1344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:13:48.371519 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:13:48.371519 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:13:48.371527 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:13:48.372836 (+ 1309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:13:48.373184 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:13:48.373185 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:13:48.373186 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:13:48.373248 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:13:48.373425 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:13:48.373426 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:13:48.373426 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:13:48.373582 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:13:48.373668 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 01:13:48.373777 (+ 109us) server_negotiation.cc:299] Negotiation successful
0926 01:13:48.373895 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 01:13:48.375685 17392 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52566:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:13:48.377256 17392 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:13:48.384076 17862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:48.379766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35633 (local address 127.0.0.1:59740)
0926 01:13:48.380208 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0926 01:13:48.380218 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 01:13:48.380272 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:13:48.380505 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:13:48.380506 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:13:48.380508 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:13:48.380547 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:13:48.380548 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:13:48.381417 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:13:48.381418 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:13:48.382817 (+ 1399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:13:48.382819 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:13:48.383773 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:13:48.383774 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:13:48.383852 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:48.383863 (+ 11us) client_negotiation.cc:769] Sending connection context
0926 01:13:48.383927 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 01:13:48.383985 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":76,"threads_started":1}
I20260926 01:13:48.384164 17864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:48.379625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32788 (local address 127.0.98.3:44299)
0926 01:13:48.379784 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0926 01:13:48.379789 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 01:13:48.380035 (+ 246us) server_negotiation.cc:407] Connection header received
0926 01:13:48.380161 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:13:48.380164 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:13:48.380181 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:13:48.380238 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:13:48.380647 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:48.381759 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:48.382963 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:48.383684 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:48.383912 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:48.383973 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 01:13:48.384037 (+ 64us) server_negotiation.cc:299] Negotiation successful
0926 01:13:48.384080 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":92,"threads_started":1}
I20260926 01:13:48.384357 17867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:48.380011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59740 (local address 127.0.98.1:35633)
0926 01:13:48.380210 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 01:13:48.380214 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 01:13:48.380275 (+ 61us) server_negotiation.cc:407] Connection header received
0926 01:13:48.380334 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:13:48.380336 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:13:48.380350 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:13:48.380424 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:13:48.380611 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:48.381318 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:48.382921 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:48.383587 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:48.383653 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:48.383733 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 01:13:48.384265 (+ 532us) server_negotiation.cc:299] Negotiation successful
0926 01:13:48.384301 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":110,"threads_started":1}
I20260926 01:13:48.384455 17866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:48.379767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54134 (local address 127.0.98.2:43985)
0926 01:13:48.379915 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0926 01:13:48.379919 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 01:13:48.380020 (+ 101us) server_negotiation.cc:407] Connection header received
0926 01:13:48.380114 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:13:48.380116 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:13:48.380133 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:13:48.380207 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:13:48.380460 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:48.381641 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:48.383135 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:48.383830 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:48.383896 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:48.383952 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 01:13:48.384296 (+ 344us) server_negotiation.cc:299] Negotiation successful
0926 01:13:48.384339 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":79,"threads_started":1}
I20260926 01:13:48.384151 17863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:48.379533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44299 (local address 127.0.0.1:32788)
0926 01:13:48.379935 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0926 01:13:48.379966 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0926 01:13:48.380066 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:13:48.380481 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:13:48.380482 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:13:48.380484 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:13:48.380537 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:13:48.380539 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:13:48.381876 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:13:48.381877 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:13:48.382856 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:13:48.382859 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:13:48.383919 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:13:48.383920 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:13:48.383975 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:48.383978 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 01:13:48.384023 (+ 45us) client_negotiation.cc:240] Negotiation successful
0926 01:13:48.384055 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":388,"mutex_wait_us":4,"thread_start_us":107,"threads_started":1}
I20260926 01:13:48.384228 17865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:48.379533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43985 (local address 127.0.0.1:54134)
0926 01:13:48.379910 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0926 01:13:48.379954 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 01:13:48.380033 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:13:48.380302 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:13:48.380305 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:13:48.380308 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:13:48.380381 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:13:48.380383 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:13:48.381756 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:13:48.381757 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:13:48.383038 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:13:48.383040 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:13:48.383988 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:13:48.383989 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:13:48.384072 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:48.384076 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 01:13:48.384124 (+ 48us) client_negotiation.cc:240] Negotiation successful
0926 01:13:48.384165 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":97,"threads_started":1}
I20260926 01:13:48.385352 17517 tablet_service.cc:1467] Processing CreateTablet for tablet c455c344dd004ce78ec7929885eddb7b (DEFAULT_TABLE table=test-table [id=86ef72e9ae3a4a8ca6a77ae75de91099]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:13:48.385398 17785 tablet_service.cc:1467] Processing CreateTablet for tablet c455c344dd004ce78ec7929885eddb7b (DEFAULT_TABLE table=test-table [id=86ef72e9ae3a4a8ca6a77ae75de91099]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:13:48.385504 17651 tablet_service.cc:1467] Processing CreateTablet for tablet c455c344dd004ce78ec7929885eddb7b (DEFAULT_TABLE table=test-table [id=86ef72e9ae3a4a8ca6a77ae75de91099]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:13:48.385725 17517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c455c344dd004ce78ec7929885eddb7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:13:48.385738 17785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c455c344dd004ce78ec7929885eddb7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:13:48.385783 17651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c455c344dd004ce78ec7929885eddb7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:13:48.388886 17868 tablet_bootstrap.cc:492] T c455c344dd004ce78ec7929885eddb7b P 79311ceddd994d968a9a3376aabe3f79: Bootstrap starting.
I20260926 01:13:48.389290 17869 tablet_bootstrap.cc:492] T c455c344dd004ce78ec7929885eddb7b P 15ea5b9f9cad47ddaecff3b98ca4c7d0: Bootstrap starting.
I20260926 01:13:48.389793 17868 tablet_bootstrap.cc:654] T c455c344dd004ce78ec7929885eddb7b P 79311ceddd994d968a9a3376aabe3f79: Neither blocks nor log segments found. Creating new log.
I20260926 01:13:48.389994 17870 tablet_bootstrap.cc:492] T c455c344dd004ce78ec7929885eddb7b P 7ce6aae72e374c7a92d1e2f7dfdf197b: Bootstrap starting.
I20260926 01:13:48.390131 17869 tablet_bootstrap.cc:654] T c455c344dd004ce78ec7929885eddb7b P 15ea5b9f9cad47ddaecff3b98ca4c7d0: Neither blocks nor log segments found. Creating new log.
I20260926 01:13:48.390439 17868 log.cc:824] T c455c344dd004ce78ec7929885eddb7b P 79311ceddd994d968a9a3376aabe3f79: Log is configured to *not* fsync() on all Append() calls
I20260926 01:13:48.390584 17870 tablet_bootstrap.cc:654] T c455c344dd004ce78ec7929885eddb7b P 7ce6aae72e374c7a92d1e2f7dfdf197b: Neither blocks nor log segments found. Creating new log.
I20260926 01:13:48.390720 17869 log.cc:824] T c455c344dd004ce78ec7929885eddb7b P 15ea5b9f9cad47ddaecff3b98ca4c7d0: Log is configured to *not* fsync() on all Append() calls
I20260926 01:13:48.391122 17870 log.cc:824] T c455c344dd004ce78ec7929885eddb7b P 7ce6aae72e374c7a92d1e2f7dfdf197b: Log is configured to *not* fsync() on all Append() calls
I20260926 01:13:48.391561 17868 tablet_bootstrap.cc:492] T c455c344dd004ce78ec7929885eddb7b P 79311ceddd994d968a9a3376aabe3f79: No bootstrap required, opened a new log
I20260926 01:13:48.391582 17869 tablet_bootstrap.cc:492] T c455c344dd004ce78ec7929885eddb7b P 15ea5b9f9cad47ddaecff3b98ca4c7d0: No bootstrap required, opened a new log
I20260926 01:13:48.391638 17869 ts_tablet_manager.cc:1397] T c455c344dd004ce78ec7929885eddb7b P 15ea5b9f9cad47ddaecff3b98ca4c7d0: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 01:13:48.391638 17868 ts_tablet_manager.cc:1397] T c455c344dd004ce78ec7929885eddb7b P 79311ceddd994d968a9a3376aabe3f79: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260926 01:13:48.393702 17869 raft_consensus.cc:348] T c455c344dd004ce78ec7929885eddb7b P 15ea5b9f9cad47ddaecff3b98ca4c7d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ea5b9f9cad47ddaecff3b98ca4c7d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44299 } } peers { permanent_uuid: "7ce6aae72e374c7a92d1e2f7dfdf197b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43985 } } peers { permanent_uuid: "79311ceddd994d968a9a3376aabe3f79" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35633 } }
I20260926 01:13:48.393702 17868 raft_consensus.cc:348] T c455c344dd004ce78ec7929885eddb7b P 79311ceddd994d968a9a3376aabe3f79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ea5b9f9cad47ddaecff3b98ca4c7d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44299 } } peers { permanent_uuid: "7ce6aae72e374c7a92d1e2f7dfdf197b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43985 } } peers { permanent_uuid: "79311ceddd994d968a9a3376aabe3f79" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35633 } }
I20260926 01:13:48.393944 17868 raft_consensus.cc:374] T c455c344dd004ce78ec7929885eddb7b P 79311ceddd994d968a9a3376aabe3f79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:13:48.393944 17869 raft_consensus.cc:374] T c455c344dd004ce78ec7929885eddb7b P 15ea5b9f9cad47ddaecff3b98ca4c7d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:13:48.393993 17868 raft_consensus.cc:729] T c455c344dd004ce78ec7929885eddb7b P 79311ceddd994d968a9a3376aabe3f79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79311ceddd994d968a9a3376aabe3f79, State: Initialized, Role: FOLLOWER
I20260926 01:13:48.393993 17869 raft_consensus.cc:729] T c455c344dd004ce78ec7929885eddb7b P 15ea5b9f9cad47ddaecff3b98ca4c7d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15ea5b9f9cad47ddaecff3b98ca4c7d0, State: Initialized, Role: FOLLOWER
I20260926 01:13:48.394124 17868 consensus_queue.cc:260] T c455c344dd004ce78ec7929885eddb7b P 79311ceddd994d968a9a3376aabe3f79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ea5b9f9cad47ddaecff3b98ca4c7d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44299 } } peers { permanent_uuid: "7ce6aae72e374c7a92d1e2f7dfdf197b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43985 } } peers { permanent_uuid: "79311ceddd994d968a9a3376aabe3f79" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35633 } }
I20260926 01:13:48.394124 17869 consensus_queue.cc:260] T c455c344dd004ce78ec7929885eddb7b P 15ea5b9f9cad47ddaecff3b98ca4c7d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ea5b9f9cad47ddaecff3b98ca4c7d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44299 } } peers { permanent_uuid: "7ce6aae72e374c7a92d1e2f7dfdf197b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43985 } } peers { permanent_uuid: "79311ceddd994d968a9a3376aabe3f79" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35633 } }
I20260926 01:13:48.394423 17869 ts_tablet_manager.cc:1428] T c455c344dd004ce78ec7929885eddb7b P 15ea5b9f9cad47ddaecff3b98ca4c7d0: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 01:13:48.394423 17868 ts_tablet_manager.cc:1428] T c455c344dd004ce78ec7929885eddb7b P 79311ceddd994d968a9a3376aabe3f79: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260926 01:13:48.394433 17583 heartbeater.cc:503] Master 127.0.98.62:35035 was elected leader, sending a full tablet report...
I20260926 01:13:48.394510 17851 heartbeater.cc:503] Master 127.0.98.62:35035 was elected leader, sending a full tablet report...
I20260926 01:13:48.395749 17870 tablet_bootstrap.cc:492] T c455c344dd004ce78ec7929885eddb7b P 7ce6aae72e374c7a92d1e2f7dfdf197b: No bootstrap required, opened a new log
I20260926 01:13:48.395833 17870 ts_tablet_manager.cc:1397] T c455c344dd004ce78ec7929885eddb7b P 7ce6aae72e374c7a92d1e2f7dfdf197b: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260926 01:13:48.397367 17870 raft_consensus.cc:348] T c455c344dd004ce78ec7929885eddb7b P 7ce6aae72e374c7a92d1e2f7dfdf197b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ea5b9f9cad47ddaecff3b98ca4c7d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44299 } } peers { permanent_uuid: "7ce6aae72e374c7a92d1e2f7dfdf197b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43985 } } peers { permanent_uuid: "79311ceddd994d968a9a3376aabe3f79" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35633 } }
I20260926 01:13:48.397540 17870 raft_consensus.cc:374] T c455c344dd004ce78ec7929885eddb7b P 7ce6aae72e374c7a92d1e2f7dfdf197b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:13:48.397565 17870 raft_consensus.cc:729] T c455c344dd004ce78ec7929885eddb7b P 7ce6aae72e374c7a92d1e2f7dfdf197b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ce6aae72e374c7a92d1e2f7dfdf197b, State: Initialized, Role: FOLLOWER
I20260926 01:13:48.397665 17870 consensus_queue.cc:260] T c455c344dd004ce78ec7929885eddb7b P 7ce6aae72e374c7a92d1e2f7dfdf197b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ea5b9f9cad47ddaecff3b98ca4c7d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44299 } } peers { permanent_uuid: "7ce6aae72e374c7a92d1e2f7dfdf197b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43985 } } peers { permanent_uuid: "79311ceddd994d968a9a3376aabe3f79" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35633 } }
I20260926 01:13:48.397953 17870 ts_tablet_manager.cc:1428] T c455c344dd004ce78ec7929885eddb7b P 7ce6aae72e374c7a92d1e2f7dfdf197b: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260926 01:13:48.397979 17717 heartbeater.cc:503] Master 127.0.98.62:35035 was elected leader, sending a full tablet report...
W20260926 01:13:48.447621 17718 tablet.cc:2369] T c455c344dd004ce78ec7929885eddb7b P 7ce6aae72e374c7a92d1e2f7dfdf197b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 01:13:48.448622 17874 raft_consensus.cc:482] T c455c344dd004ce78ec7929885eddb7b P 15ea5b9f9cad47ddaecff3b98ca4c7d0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:13:48.448727 17874 raft_consensus.cc:504] T c455c344dd004ce78ec7929885eddb7b P 15ea5b9f9cad47ddaecff3b98ca4c7d0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ea5b9f9cad47ddaecff3b98ca4c7d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44299 } } peers { permanent_uuid: "7ce6aae72e374c7a92d1e2f7dfdf197b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43985 } } peers { permanent_uuid: "79311ceddd994d968a9a3376aabe3f79" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35633 } }
I20260926 01:13:48.449061 17874 leader_election.cc:288] T c455c344dd004ce78ec7929885eddb7b P 15ea5b9f9cad47ddaecff3b98ca4c7d0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7ce6aae72e374c7a92d1e2f7dfdf197b (127.0.98.2:43985), 79311ceddd994d968a9a3376aabe3f79 (127.0.98.1:35633)
I20260926 01:13:48.451915 17866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:48.449363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51147 (local address 127.0.98.2:43985)
0926 01:13:48.449412 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 01:13:48.449414 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:13:48.449424 (+ 10us) server_negotiation.cc:407] Connection header received
0926 01:13:48.449455 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:13:48.449456 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:13:48.449468 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:13:48.449540 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:13:48.449707 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:48.450226 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:48.451225 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:48.451602 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:48.451662 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:48.451674 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0926 01:13:48.451810 (+ 136us) server_negotiation.cc:299] Negotiation successful
0926 01:13:48.451849 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 01:13:48.451922 17867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:48.449437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53431 (local address 127.0.98.1:35633)
0926 01:13:48.449476 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 01:13:48.449478 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:13:48.449590 (+ 112us) server_negotiation.cc:407] Connection header received
0926 01:13:48.449620 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:13:48.449622 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:13:48.449634 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:13:48.449680 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:13:48.449871 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:48.450339 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:48.451245 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:48.451603 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:48.451662 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:48.451674 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0926 01:13:48.451817 (+ 143us) server_negotiation.cc:299] Negotiation successful
0926 01:13:48.451855 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 01:13:48.451920 17853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:48.449266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43985 (local address 127.0.98.3:51147)
0926 01:13:48.449316 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0926 01:13:48.449328 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 01:13:48.449376 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:13:48.449580 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:13:48.449581 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:13:48.449583 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:13:48.449646 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:13:48.449648 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:13:48.450329 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:13:48.450330 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:13:48.451129 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:13:48.451132 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:13:48.451674 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:13:48.451675 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:13:48.451748 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:48.451752 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 01:13:48.451796 (+ 44us) client_negotiation.cc:240] Negotiation successful
0926 01:13:48.451837 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260926 01:13:48.451929 17877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:48.449364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35633 (local address 127.0.98.3:53431)
0926 01:13:48.449522 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 01:13:48.449529 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0926 01:13:48.449576 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:13:48.449717 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:13:48.449718 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:13:48.449720 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:13:48.449785 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:13:48.449786 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:13:48.450408 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:13:48.450409 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:13:48.451161 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:13:48.451164 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:13:48.451678 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:13:48.451679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:13:48.451755 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:48.451759 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 01:13:48.451803 (+ 44us) client_negotiation.cc:240] Negotiation successful
0926 01:13:48.451843 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":120,"threads_started":1}
I20260926 01:13:48.452329 17671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c455c344dd004ce78ec7929885eddb7b" candidate_uuid: "15ea5b9f9cad47ddaecff3b98ca4c7d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ce6aae72e374c7a92d1e2f7dfdf197b" is_pre_election: true
I20260926 01:13:48.452327 17537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c455c344dd004ce78ec7929885eddb7b" candidate_uuid: "15ea5b9f9cad47ddaecff3b98ca4c7d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79311ceddd994d968a9a3376aabe3f79" is_pre_election: true
I20260926 01:13:48.452484 17537 raft_consensus.cc:2454] T c455c344dd004ce78ec7929885eddb7b P 79311ceddd994d968a9a3376aabe3f79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15ea5b9f9cad47ddaecff3b98ca4c7d0 in term 0.
I20260926 01:13:48.452493 17671 raft_consensus.cc:2454] T c455c344dd004ce78ec7929885eddb7b P 7ce6aae72e374c7a92d1e2f7dfdf197b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15ea5b9f9cad47ddaecff3b98ca4c7d0 in term 0.
I20260926 01:13:48.452718 17741 leader_election.cc:302] T c455c344dd004ce78ec7929885eddb7b P 15ea5b9f9cad47ddaecff3b98ca4c7d0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15ea5b9f9cad47ddaecff3b98ca4c7d0, 79311ceddd994d968a9a3376aabe3f79; no voters:
I20260926 01:13:48.452842 17874 raft_consensus.cc:2781] T c455c344dd004ce78ec7929885eddb7b P 15ea5b9f9cad47ddaecff3b98ca4c7d0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:13:48.452893 17874 raft_consensus.cc:482] T c455c344dd004ce78ec7929885eddb7b P 15ea5b9f9cad47ddaecff3b98ca4c7d0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:13:48.452915 17874 raft_consensus.cc:3037] T c455c344dd004ce78ec7929885eddb7b P 15ea5b9f9cad47ddaecff3b98ca4c7d0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:13:48.453588 17874 raft_consensus.cc:504] T c455c344dd004ce78ec7929885eddb7b P 15ea5b9f9cad47ddaecff3b98ca4c7d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ea5b9f9cad47ddaecff3b98ca4c7d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44299 } } peers { permanent_uuid: "7ce6aae72e374c7a92d1e2f7dfdf197b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43985 } } peers { permanent_uuid: "79311ceddd994d968a9a3376aabe3f79" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35633 } }
I20260926 01:13:48.453722 17874 leader_election.cc:288] T c455c344dd004ce78ec7929885eddb7b P 15ea5b9f9cad47ddaecff3b98ca4c7d0 [CANDIDATE]: Term 1 election: Requested vote from peers 7ce6aae72e374c7a92d1e2f7dfdf197b (127.0.98.2:43985), 79311ceddd994d968a9a3376aabe3f79 (127.0.98.1:35633)
I20260926 01:13:48.453949 17671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c455c344dd004ce78ec7929885eddb7b" candidate_uuid: "15ea5b9f9cad47ddaecff3b98ca4c7d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ce6aae72e374c7a92d1e2f7dfdf197b"
I20260926 01:13:48.453972 17537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c455c344dd004ce78ec7929885eddb7b" candidate_uuid: "15ea5b9f9cad47ddaecff3b98ca4c7d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79311ceddd994d968a9a3376aabe3f79"
I20260926 01:13:48.454025 17671 raft_consensus.cc:3037] T c455c344dd004ce78ec7929885eddb7b P 7ce6aae72e374c7a92d1e2f7dfdf197b [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:13:48.454042 17537 raft_consensus.cc:3037] T c455c344dd004ce78ec7929885eddb7b P 79311ceddd994d968a9a3376aabe3f79 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:13:48.454702 17671 raft_consensus.cc:2454] T c455c344dd004ce78ec7929885eddb7b P 7ce6aae72e374c7a92d1e2f7dfdf197b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15ea5b9f9cad47ddaecff3b98ca4c7d0 in term 1.
I20260926 01:13:48.454764 17537 raft_consensus.cc:2454] T c455c344dd004ce78ec7929885eddb7b P 79311ceddd994d968a9a3376aabe3f79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15ea5b9f9cad47ddaecff3b98ca4c7d0 in term 1.
I20260926 01:13:48.454918 17741 leader_election.cc:302] T c455c344dd004ce78ec7929885eddb7b P 15ea5b9f9cad47ddaecff3b98ca4c7d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15ea5b9f9cad47ddaecff3b98ca4c7d0, 79311ceddd994d968a9a3376aabe3f79; no voters:
I20260926 01:13:48.455045 17874 raft_consensus.cc:2781] T c455c344dd004ce78ec7929885eddb7b P 15ea5b9f9cad47ddaecff3b98ca4c7d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:13:48.455255 17874 raft_consensus.cc:686] T c455c344dd004ce78ec7929885eddb7b P 15ea5b9f9cad47ddaecff3b98ca4c7d0 [term 1 LEADER]: Becoming Leader. State: Replica: 15ea5b9f9cad47ddaecff3b98ca4c7d0, State: Running, Role: LEADER
I20260926 01:13:48.455358 17874 consensus_queue.cc:237] T c455c344dd004ce78ec7929885eddb7b P 15ea5b9f9cad47ddaecff3b98ca4c7d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ea5b9f9cad47ddaecff3b98ca4c7d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44299 } } peers { permanent_uuid: "7ce6aae72e374c7a92d1e2f7dfdf197b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43985 } } peers { permanent_uuid: "79311ceddd994d968a9a3376aabe3f79" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35633 } }
I20260926 01:13:48.456326 17392 catalog_manager.cc:5495] T c455c344dd004ce78ec7929885eddb7b P 15ea5b9f9cad47ddaecff3b98ca4c7d0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 15ea5b9f9cad47ddaecff3b98ca4c7d0 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "15ea5b9f9cad47ddaecff3b98ca4c7d0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ea5b9f9cad47ddaecff3b98ca4c7d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44299 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ce6aae72e374c7a92d1e2f7dfdf197b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43985 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "79311ceddd994d968a9a3376aabe3f79" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35633 } health_report { overall_health: UNKNOWN } } }
I20260926 01:13:48.475977 17864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:48.473764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32790 (local address 127.0.98.3:44299)
0926 01:13:48.473870 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0926 01:13:48.473872 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:13:48.473883 (+ 11us) server_negotiation.cc:407] Connection header received
0926 01:13:48.473925 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:13:48.473926 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:13:48.473948 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:13:48.474017 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:13:48.474117 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:48.474589 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:48.475301 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:48.475484 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:48.475530 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:48.475647 (+ 117us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:13:48.475785 (+ 138us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:13:48.475845 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 01:13:48.475860 (+ 15us) server_negotiation.cc:299] Negotiation successful
0926 01:13:48.475878 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 01:13:48.477716 17537 raft_consensus.cc:1261] T c455c344dd004ce78ec7929885eddb7b P 79311ceddd994d968a9a3376aabe3f79 [term 1 FOLLOWER]: Refusing update from remote peer 15ea5b9f9cad47ddaecff3b98ca4c7d0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:13:48.477725 17671 raft_consensus.cc:1261] T c455c344dd004ce78ec7929885eddb7b P 7ce6aae72e374c7a92d1e2f7dfdf197b [term 1 FOLLOWER]: Refusing update from remote peer 15ea5b9f9cad47ddaecff3b98ca4c7d0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:13:48.478081 17874 consensus_queue.cc:1035] T c455c344dd004ce78ec7929885eddb7b P 15ea5b9f9cad47ddaecff3b98ca4c7d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79311ceddd994d968a9a3376aabe3f79" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:13:48.478240 17878 consensus_queue.cc:1035] T c455c344dd004ce78ec7929885eddb7b P 15ea5b9f9cad47ddaecff3b98ca4c7d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ce6aae72e374c7a92d1e2f7dfdf197b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:13:48.480300 17881 mvcc.cc:205] Tried to move back new op lower bound from 7333417895842263040 to 7333417895754219520. Current Snapshot: MvccSnapshot[applied={T|T < 7333417895842263040}]
I20260926 01:13:48.481003 17882 mvcc.cc:205] Tried to move back new op lower bound from 7333417895842263040 to 7333417895754219520. Current Snapshot: MvccSnapshot[applied={T|T < 7333417895842263040 or (T in {7333417895842263040})}]
I20260926 01:13:48.482645 17391 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52566:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:13:48.482729 17391 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52566:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:13:48.483425 17391 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0ed585b0113543ffba64a64ec6dd9c62: Sending DeleteTablet for 3 replicas of tablet c455c344dd004ce78ec7929885eddb7b
I20260926 01:13:48.483749 17517 tablet_service.cc:1514] Processing DeleteTablet for tablet c455c344dd004ce78ec7929885eddb7b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:13:48 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59740
I20260926 01:13:48.483757 17651 tablet_service.cc:1514] Processing DeleteTablet for tablet c455c344dd004ce78ec7929885eddb7b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:13:48 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54134
I20260926 01:13:48.483804 17785 tablet_service.cc:1514] Processing DeleteTablet for tablet c455c344dd004ce78ec7929885eddb7b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:13:48 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:32788
I20260926 01:13:48.483955 17890 tablet_replica.cc:330] T c455c344dd004ce78ec7929885eddb7b P 79311ceddd994d968a9a3376aabe3f79: stopping tablet replica
I20260926 01:13:48.483973 17891 tablet_replica.cc:330] T c455c344dd004ce78ec7929885eddb7b P 7ce6aae72e374c7a92d1e2f7dfdf197b: stopping tablet replica
I20260926 01:13:48.484055 17890 raft_consensus.cc:2229] T c455c344dd004ce78ec7929885eddb7b P 79311ceddd994d968a9a3376aabe3f79 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:13:48.484055 17891 raft_consensus.cc:2229] T c455c344dd004ce78ec7929885eddb7b P 7ce6aae72e374c7a92d1e2f7dfdf197b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:13:48.484153 17890 raft_consensus.cc:2258] T c455c344dd004ce78ec7929885eddb7b P 79311ceddd994d968a9a3376aabe3f79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:13:48.484177 17891 raft_consensus.cc:2258] T c455c344dd004ce78ec7929885eddb7b P 7ce6aae72e374c7a92d1e2f7dfdf197b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:13:48.484550 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 17454
I20260926 01:13:48.484577 17891 ts_tablet_manager.cc:1905] T c455c344dd004ce78ec7929885eddb7b P 7ce6aae72e374c7a92d1e2f7dfdf197b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:13:48.485278 17892 tablet_replica.cc:330] T c455c344dd004ce78ec7929885eddb7b P 15ea5b9f9cad47ddaecff3b98ca4c7d0: stopping tablet replica
I20260926 01:13:48.485425 17892 raft_consensus.cc:2229] T c455c344dd004ce78ec7929885eddb7b P 15ea5b9f9cad47ddaecff3b98ca4c7d0 [term 1 LEADER]: Raft consensus shutting down.
I20260926 01:13:48.485587 17892 raft_consensus.cc:2258] T c455c344dd004ce78ec7929885eddb7b P 15ea5b9f9cad47ddaecff3b98ca4c7d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:13:48.486140 17891 ts_tablet_manager.cc:1918] T c455c344dd004ce78ec7929885eddb7b P 7ce6aae72e374c7a92d1e2f7dfdf197b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:13:48.486214 17891 log.cc:1196] T c455c344dd004ce78ec7929885eddb7b P 7ce6aae72e374c7a92d1e2f7dfdf197b: Deleting WAL directory at /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384993574762-392-0/minicluster-data/ts-1/wal/wals/c455c344dd004ce78ec7929885eddb7b
I20260926 01:13:48.486466 17891 ts_tablet_manager.cc:1939] T c455c344dd004ce78ec7929885eddb7b P 7ce6aae72e374c7a92d1e2f7dfdf197b: Deleting consensus metadata
I20260926 01:13:48.487131 17379 catalog_manager.cc:4841] TS 7ce6aae72e374c7a92d1e2f7dfdf197b (127.0.98.2:43985): tablet c455c344dd004ce78ec7929885eddb7b (table test-table [id=86ef72e9ae3a4a8ca6a77ae75de91099]) successfully deleted
I20260926 01:13:48.490284 17892 ts_tablet_manager.cc:1905] T c455c344dd004ce78ec7929885eddb7b P 15ea5b9f9cad47ddaecff3b98ca4c7d0: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 01:13:48.491613 17379 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35633 (error 108)
I20260926 01:13:48.491644 17892 ts_tablet_manager.cc:1918] T c455c344dd004ce78ec7929885eddb7b P 15ea5b9f9cad47ddaecff3b98ca4c7d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:13:48.491807 17892 log.cc:1196] T c455c344dd004ce78ec7929885eddb7b P 15ea5b9f9cad47ddaecff3b98ca4c7d0: Deleting WAL directory at /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384993574762-392-0/minicluster-data/ts-2/wal/wals/c455c344dd004ce78ec7929885eddb7b
I20260926 01:13:48.491801 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 17588
I20260926 01:13:48.492069 17892 ts_tablet_manager.cc:1939] T c455c344dd004ce78ec7929885eddb7b P 15ea5b9f9cad47ddaecff3b98ca4c7d0: Deleting consensus metadata
I20260926 01:13:48.492846 17378 catalog_manager.cc:4841] TS 15ea5b9f9cad47ddaecff3b98ca4c7d0 (127.0.98.3:44299): tablet c455c344dd004ce78ec7929885eddb7b (table test-table [id=86ef72e9ae3a4a8ca6a77ae75de91099]) successfully deleted
I20260926 01:13:48.494761 17865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:48.494592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35633 (local address 127.0.0.1:59756)
0926 01:13:48.494669 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0926 01:13:48.494726 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260926 01:13:48.496546 17379 catalog_manager.cc:4584] TS 79311ceddd994d968a9a3376aabe3f79 (127.0.98.1:35633): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c455c344dd004ce78ec7929885eddb7b: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35633: connect: Connection refused (error 111)
I20260926 01:13:48.496896 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 17722
I20260926 01:13:48.503433 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 17360
2026-09-26T01:13:48Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (1887 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-26T01:13:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:13:48Z Disabled control of system clock
I20260926 01:13:48.529804 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384993574762-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384993574762-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384993574762-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39787
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:38947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39787
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 01:13:48.611809 17897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:13:48.612071 17897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:13:48.612093 17897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:13:48.613333 17897 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:13:48.613379 17897 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:13:48.613389 17897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:13:48.613400 17897 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:13:48.613411 17897 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:13:48.614917 17897 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384993574762-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384993574762-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39787
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39787
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17897
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384993574762-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:13:48.615156 17897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:13:48.615442 17897 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:13:48.620386 17902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:13:48.620374 17903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:13:48.620374 17905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:13:48.620827 17897 server_base.cc:1034] running on GCE node
I20260926 01:13:48.621098 17897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:13:48.621409 17897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:13:48.622553 17897 hybrid_clock.cc:648] HybridClock initialized: now 1790385228622516 us; error 51 us; skew 500 ppm
I20260926 01:13:48.624032 17897 webserver.cc:460] Webserver started at http://127.0.98.62:46191/ using document root <none> and password file <none>
I20260926 01:13:48.624276 17897 fs_manager.cc:359] Metadata directory not provided
I20260926 01:13:48.624325 17897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:13:48.624470 17897 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:13:48.625545 17897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384993574762-392-0/minicluster-data/master-0/data/instance:
uuid: "21659de43a0f423bb95e88e366c675bd"
format_stamp: "Formatted at 2026-09-26 01:13:48 on dist-test-slave-7b22"
server_key: "dd39b9132d2fc6fc09cb400123048698"
server_key_iv: "d493c2fe79fd1ab531c869968cbe6bf9"
server_key_version: "encryptionkey@0"
I20260926 01:13:48.625931 17897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384993574762-392-0/minicluster-data/master-0/wal/instance:
uuid: "21659de43a0f423bb95e88e366c675bd"
format_stamp: "Formatted at 2026-09-26 01:13:48 on dist-test-slave-7b22"
server_key: "dd39b9132d2fc6fc09cb400123048698"
server_key_iv: "d493c2fe79fd1ab531c869968cbe6bf9"
server_key_version: "encryptionkey@0"
I20260926 01:13:48.627292 17897 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 01:13:48.628132 17911 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:13:48.628301 17897 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 01:13:48.628389 17897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384993574762-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384993574762-392-0/minicluster-data/master-0/wal
uuid: "21659de43a0f423bb95e88e366c675bd"
format_stamp: "Formatted at 2026-09-26 01:13:48 on dist-test-slave-7b22"
server_key: "dd39b9132d2fc6fc09cb400123048698"
server_key_iv: "d493c2fe79fd1ab531c869968cbe6bf9"
server_key_version: "encryptionkey@0"
I20260926 01:13:48.628464 17897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384993574762-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384993574762-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384993574762-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:13:48.645537 17897 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:13:48.645928 17897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:13:48.646067 17897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:13:48.649693 17897 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39787
I20260926 01:13:48.649755 17963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39787 every 8 connection(s)
I20260926 01:13:48.650108 17897 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
I20260926 01:13:48.651134 17964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:13:48.653995 17964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21659de43a0f423bb95e88e366c675bd: Bootstrap starting.
I20260926 01:13:48.654402 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 17897
I20260926 01:13:48.654505 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384993574762-392-0/minicluster-data/master-0/wal/instance
I20260926 01:13:48.654645 17964 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 21659de43a0f423bb95e88e366c675bd: Neither blocks nor log segments found. Creating new log.
I20260926 01:13:48.654697 392 external_mini_cluster.cc:1428] Setting key f71393390705ecd623e16a2b092eacb2
I20260926 01:13:48.654970 17964 log.cc:824] T 00000000000000000000000000000000 P 21659de43a0f423bb95e88e366c675bd: Log is configured to *not* fsync() on all Append() calls
I20260926 01:13:48.655805 17964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21659de43a0f423bb95e88e366c675bd: No bootstrap required, opened a new log
I20260926 01:13:48.658070 17964 raft_consensus.cc:348] T 00000000000000000000000000000000 P 21659de43a0f423bb95e88e366c675bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21659de43a0f423bb95e88e366c675bd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39787 } }
I20260926 01:13:48.658222 17964 raft_consensus.cc:374] T 00000000000000000000000000000000 P 21659de43a0f423bb95e88e366c675bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:13:48.658254 17964 raft_consensus.cc:729] T 00000000000000000000000000000000 P 21659de43a0f423bb95e88e366c675bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21659de43a0f423bb95e88e366c675bd, State: Initialized, Role: FOLLOWER
I20260926 01:13:48.658381 17964 consensus_queue.cc:260] T 00000000000000000000000000000000 P 21659de43a0f423bb95e88e366c675bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21659de43a0f423bb95e88e366c675bd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39787 } }
I20260926 01:13:48.658454 17964 raft_consensus.cc:388] T 00000000000000000000000000000000 P 21659de43a0f423bb95e88e366c675bd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:13:48.658483 17964 raft_consensus.cc:482] T 00000000000000000000000000000000 P 21659de43a0f423bb95e88e366c675bd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:13:48.658531 17964 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 21659de43a0f423bb95e88e366c675bd [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:13:48.659128 17964 raft_consensus.cc:504] T 00000000000000000000000000000000 P 21659de43a0f423bb95e88e366c675bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21659de43a0f423bb95e88e366c675bd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39787 } }
I20260926 01:13:48.659224 17964 leader_election.cc:302] T 00000000000000000000000000000000 P 21659de43a0f423bb95e88e366c675bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 21659de43a0f423bb95e88e366c675bd; no voters:
I20260926 01:13:48.659425 17964 leader_election.cc:288] T 00000000000000000000000000000000 P 21659de43a0f423bb95e88e366c675bd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:13:48.659487 17969 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 21659de43a0f423bb95e88e366c675bd [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:13:48.659627 17969 raft_consensus.cc:686] T 00000000000000000000000000000000 P 21659de43a0f423bb95e88e366c675bd [term 1 LEADER]: Becoming Leader. State: Replica: 21659de43a0f423bb95e88e366c675bd, State: Running, Role: LEADER
I20260926 01:13:48.659735 17964 sys_catalog.cc:564] T 00000000000000000000000000000000 P 21659de43a0f423bb95e88e366c675bd [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:13:48.659775 17969 consensus_queue.cc:237] T 00000000000000000000000000000000 P 21659de43a0f423bb95e88e366c675bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21659de43a0f423bb95e88e366c675bd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39787 } }
I20260926 01:13:48.660382 17971 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21659de43a0f423bb95e88e366c675bd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "21659de43a0f423bb95e88e366c675bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21659de43a0f423bb95e88e366c675bd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39787 } } }
I20260926 01:13:48.660396 17970 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21659de43a0f423bb95e88e366c675bd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 21659de43a0f423bb95e88e366c675bd. Latest consensus state: current_term: 1 leader_uuid: "21659de43a0f423bb95e88e366c675bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21659de43a0f423bb95e88e366c675bd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39787 } } }
I20260926 01:13:48.660687 17971 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21659de43a0f423bb95e88e366c675bd [sys.catalog]: This master's current role is: LEADER
I20260926 01:13:48.660699 17970 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21659de43a0f423bb95e88e366c675bd [sys.catalog]: This master's current role is: LEADER
I20260926 01:13:48.660845 17968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:48.655292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39274 (local address 127.0.98.62:39787)
0926 01:13:48.655527 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0926 01:13:48.655534 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 01:13:48.655560 (+ 26us) server_negotiation.cc:407] Connection header received
0926 01:13:48.656065 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:13:48.656072 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:13:48.656136 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:13:48.656260 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:13:48.656517 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:48.657745 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:48.658640 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:48.659084 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:48.659199 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:48.659975 (+ 776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:13:48.659978 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:13:48.659979 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:13:48.660184 (+ 205us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:13:48.660237 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:13:48.660254 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:13:48.660317 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 01:13:48.660504 (+ 187us) server_negotiation.cc:299] Negotiation successful
0926 01:13:48.660601 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":105,"threads_started":1}
W20260926 01:13:48.661690 17984 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 21659de43a0f423bb95e88e366c675bd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 01:13:48.661734 17984 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 01:13:48.661790 17979 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:13:48.661994 17979 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:13:48.663401 17979 catalog_manager.cc:1347] Generated new cluster ID: 46dc557c01254a94bfedb6411e3a0988
I20260926 01:13:48.663458 17979 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 01:13:48.673782 17979 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:13:48.674566 17979 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:13:48.696273 17979 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 21659de43a0f423bb95e88e366c675bd: Generated new TSK 0
I20260926 01:13:48.696509 17979 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 01:13:48.698422 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384993574762-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384993574762-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384993574762-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384993574762-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39787
--builtin_ntp_servers=127.0.98.20:38947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 01:13:48.782133 17988 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:13:48.782366 17988 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:13:48.782397 17988 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:13:48.783643 17988 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:13:48.783694 17988 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:13:48.783742 17988 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 01:13:48.785228 17988 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384993574762-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384993574762-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384993574762-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39787
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17988
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384993574762-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:13:48.785506 17988 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:13:48.785848 17988 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:13:48.790967 17996 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:13:48.790970 17993 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:13:48.791091 17994 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:13:48.791328 17988 server_base.cc:1034] running on GCE node
I20260926 01:13:48.791569 17988 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:13:48.791815 17988 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:13:48.792990 17988 hybrid_clock.cc:648] HybridClock initialized: now 1790385228792976 us; error 34 us; skew 500 ppm
I20260926 01:13:48.794276 17988 webserver.cc:460] Webserver started at http://127.0.98.1:34921/ using document root <none> and password file <none>
I20260926 01:13:48.794508 17988 fs_manager.cc:359] Metadata directory not provided
I20260926 01:13:48.794555 17988 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:13:48.794688 17988 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:13:48.795758 17988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384993574762-392-0/minicluster-data/ts-0/data/instance:
uuid: "56e13026ccd5462f85b8425460b95c6c"
format_stamp: "Formatted at 2026-09-26 01:13:48 on dist-test-slave-7b22"
server_key: "e4c686ed7713137ae4ec173f140e538d"
server_key_iv: "dfeb759010e51a7bdd32faef044819cd"
server_key_version: "encryptionkey@0"
I20260926 01:13:48.796075 17988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384993574762-392-0/minicluster-data/ts-0/wal/instance:
uuid: "56e13026ccd5462f85b8425460b95c6c"
format_stamp: "Formatted at 2026-09-26 01:13:48 on dist-test-slave-7b22"
server_key: "e4c686ed7713137ae4ec173f140e538d"
server_key_iv: "dfeb759010e51a7bdd32faef044819cd"
server_key_version: "encryptionkey@0"
I20260926 01:13:48.797371 17988 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 01:13:48.798295 18002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:13:48.798547 17988 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 01:13:48.798638 17988 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384993574762-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384993574762-392-0/minicluster-data/ts-0/wal
uuid: "56e13026ccd5462f85b8425460b95c6c"
format_stamp: "Formatted at 2026-09-26 01:13:48 on dist-test-slave-7b22"
server_key: "e4c686ed7713137ae4ec173f140e538d"
server_key_iv: "dfeb759010e51a7bdd32faef044819cd"
server_key_version: "encryptionkey@0"
I20260926 01:13:48.798718 17988 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384993574762-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384993574762-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384993574762-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:13:48.830201 17988 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:13:48.830541 17988 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:13:48.830672 17988 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:13:48.830889 17988 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:13:48.831238 17988 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:13:48.831291 17988 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:13:48.831323 17988 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:13:48.831350 17988 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:13:48.836822 17988 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39677
I20260926 01:13:48.836870 18115 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39677 every 8 connection(s)
I20260926 01:13:48.837204 17988 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384993574762-392-0/minicluster-data/ts-0/data/info.pb
I20260926 01:13:48.842460 17968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:48.837939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38909 (local address 127.0.98.62:39787)
0926 01:13:48.837989 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 01:13:48.837991 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:13:48.838210 (+ 219us) server_negotiation.cc:407] Connection header received
0926 01:13:48.838701 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:13:48.838703 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:13:48.838714 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:13:48.838776 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:13:48.839422 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:48.839930 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:48.840941 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:48.841167 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:48.841239 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:48.842123 (+ 884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:13:48.842124 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:13:48.842124 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:13:48.842129 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:13:48.842144 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:13:48.842148 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:13:48.842201 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 01:13:48.842362 (+ 161us) server_negotiation.cc:299] Negotiation successful
0926 01:13:48.842391 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 01:13:48.842583 18118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:48.837905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39787 (local address 127.0.98.1:38909)
0926 01:13:48.838081 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 01:13:48.838131 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 01:13:48.838602 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:13:48.838828 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:13:48.838831 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:13:48.838840 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:13:48.839341 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:13:48.839344 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:13:48.840001 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:13:48.840002 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:13:48.840805 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:13:48.840807 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:13:48.841250 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:13:48.841251 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:13:48.841367 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:48.841982 (+ 615us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 01:13:48.841982 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:13:48.842011 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 01:13:48.842035 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:13:48.842236 (+ 201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:13:48.842243 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:13:48.842246 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 01:13:48.842320 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 01:13:48.842411 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":67,"threads_started":1}
I20260926 01:13:48.843086 18116 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39787
I20260926 01:13:48.843183 18116 heartbeater.cc:464] Registering TS with master...
I20260926 01:13:48.843345 18116 heartbeater.cc:511] Master 127.0.98.62:39787 requested a full tablet report, sending...
I20260926 01:13:48.843815 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 17988
I20260926 01:13:48.843918 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384993574762-392-0/minicluster-data/ts-0/wal/instance
I20260926 01:13:48.843927 17928 ts_manager.cc:194] Registered new tserver with Master: 56e13026ccd5462f85b8425460b95c6c (127.0.98.1:39677)
I20260926 01:13:48.844113 392 external_mini_cluster.cc:1428] Setting key ceecacc75d393950cec63d153e2479a7
I20260926 01:13:48.844684 17928 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:38909
I20260926 01:13:48.845194 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384993574762-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384993574762-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384993574762-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384993574762-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39787
--builtin_ntp_servers=127.0.98.20:38947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 01:13:48.926712 18119 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:13:48.926959 18119 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:13:48.926985 18119 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:13:48.928176 18119 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:13:48.928217 18119 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:13:48.928246 18119 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 01:13:48.929661 18119 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384993574762-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384993574762-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384993574762-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39787
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18119
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384993574762-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:13:48.929916 18119 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:13:48.930194 18119 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:13:48.935328 18127 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:13:48.935336 18124 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:13:48.935514 18119 server_base.cc:1034] running on GCE node
W20260926 01:13:48.935328 18125 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:13:48.935788 18119 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:13:48.936084 18119 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:13:48.937263 18119 hybrid_clock.cc:648] HybridClock initialized: now 1790385228937247 us; error 31 us; skew 500 ppm
I20260926 01:13:48.938815 18119 webserver.cc:460] Webserver started at http://127.0.98.2:35979/ using document root <none> and password file <none>
I20260926 01:13:48.939054 18119 fs_manager.cc:359] Metadata directory not provided
I20260926 01:13:48.939137 18119 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:13:48.939273 18119 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:13:48.940399 18119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384993574762-392-0/minicluster-data/ts-1/data/instance:
uuid: "e26dffe37b4c4797bf680d9a425fd4cd"
format_stamp: "Formatted at 2026-09-26 01:13:48 on dist-test-slave-7b22"
server_key: "8c12685d620df2fc4b73a3c0a2800526"
server_key_iv: "ab5fad6be1c57201730032497b0cb8f3"
server_key_version: "encryptionkey@0"
I20260926 01:13:48.940725 18119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384993574762-392-0/minicluster-data/ts-1/wal/instance:
uuid: "e26dffe37b4c4797bf680d9a425fd4cd"
format_stamp: "Formatted at 2026-09-26 01:13:48 on dist-test-slave-7b22"
server_key: "8c12685d620df2fc4b73a3c0a2800526"
server_key_iv: "ab5fad6be1c57201730032497b0cb8f3"
server_key_version: "encryptionkey@0"
I20260926 01:13:48.942095 18119 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 01:13:48.942927 18133 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:13:48.943112 18119 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:13:48.943197 18119 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384993574762-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384993574762-392-0/minicluster-data/ts-1/wal
uuid: "e26dffe37b4c4797bf680d9a425fd4cd"
format_stamp: "Formatted at 2026-09-26 01:13:48 on dist-test-slave-7b22"
server_key: "8c12685d620df2fc4b73a3c0a2800526"
server_key_iv: "ab5fad6be1c57201730032497b0cb8f3"
server_key_version: "encryptionkey@0"
I20260926 01:13:48.943277 18119 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384993574762-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384993574762-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384993574762-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:13:48.960999 18119 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:13:48.961323 18119 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:13:48.961457 18119 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:13:48.961719 18119 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:13:48.962071 18119 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:13:48.962108 18119 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:13:48.962137 18119 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:13:48.962154 18119 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:13:48.967448 18119 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40797
I20260926 01:13:48.967519 18246 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40797 every 8 connection(s)
I20260926 01:13:48.967837 18119 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384993574762-392-0/minicluster-data/ts-1/data/info.pb
I20260926 01:13:48.970329 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 18119
I20260926 01:13:48.970435 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384993574762-392-0/minicluster-data/ts-1/wal/instance
I20260926 01:13:48.970629 392 external_mini_cluster.cc:1428] Setting key a63842774827d8d6615989ea88aa2f0c
I20260926 01:13:48.971585 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384993574762-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384993574762-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384993574762-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384993574762-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39787
--builtin_ntp_servers=127.0.98.20:38947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 01:13:48.973997 17968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:48.968581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36915 (local address 127.0.98.62:39787)
0926 01:13:48.968635 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 01:13:48.968637 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:13:48.968945 (+ 308us) server_negotiation.cc:407] Connection header received
0926 01:13:48.969429 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:13:48.969430 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:13:48.969442 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:13:48.969500 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:13:48.970339 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:48.971107 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:48.972239 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:48.972501 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:48.972574 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:48.973644 (+ 1070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:13:48.973685 (+ 41us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:13:48.973685 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:13:48.973690 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:13:48.973706 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:13:48.973710 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:13:48.973760 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 01:13:48.973900 (+ 140us) server_negotiation.cc:299] Negotiation successful
0926 01:13:48.973933 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 01:13:48.974223 18249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:48.968558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39787 (local address 127.0.98.2:36915)
0926 01:13:48.968796 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0926 01:13:48.968835 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 01:13:48.969336 (+ 501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:13:48.969572 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:13:48.969575 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:13:48.969583 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:13:48.970223 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:13:48.970229 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:13:48.971193 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:13:48.971194 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:13:48.972156 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:13:48.972159 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:13:48.972654 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:13:48.972656 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:13:48.972845 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:48.973462 (+ 617us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 01:13:48.973463 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:13:48.973503 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 01:13:48.973534 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:13:48.973779 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:13:48.973784 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:13:48.973787 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 01:13:48.973881 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 01:13:48.973996 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":81,"threads_started":1}
I20260926 01:13:48.974721 18247 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39787
I20260926 01:13:48.974820 18247 heartbeater.cc:464] Registering TS with master...
I20260926 01:13:48.975003 18247 heartbeater.cc:511] Master 127.0.98.62:39787 requested a full tablet report, sending...
I20260926 01:13:48.975508 17928 ts_manager.cc:194] Registered new tserver with Master: e26dffe37b4c4797bf680d9a425fd4cd (127.0.98.2:40797)
I20260926 01:13:48.975870 17928 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:36915
W20260926 01:13:49.055379 18250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:13:49.055610 18250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:13:49.055641 18250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:13:49.056895 18250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:13:49.056947 18250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:13:49.056986 18250 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 01:13:49.058461 18250 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384993574762-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384993574762-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384993574762-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39787
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18250
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384993574762-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:13:49.058717 18250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:13:49.059036 18250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:13:49.064297 18258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:13:49.064297 18256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:13:49.064347 18255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:13:49.064495 18250 server_base.cc:1034] running on GCE node
I20260926 01:13:49.064754 18250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:13:49.065021 18250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:13:49.066184 18250 hybrid_clock.cc:648] HybridClock initialized: now 1790385229066168 us; error 29 us; skew 500 ppm
I20260926 01:13:49.067720 18250 webserver.cc:460] Webserver started at http://127.0.98.3:38127/ using document root <none> and password file <none>
I20260926 01:13:49.067953 18250 fs_manager.cc:359] Metadata directory not provided
I20260926 01:13:49.068004 18250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:13:49.068135 18250 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:13:49.069351 18250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384993574762-392-0/minicluster-data/ts-2/data/instance:
uuid: "f0035afb0aef4c888bff975c35b48794"
format_stamp: "Formatted at 2026-09-26 01:13:49 on dist-test-slave-7b22"
server_key: "ceda051d092078b891505bf9cfda06c0"
server_key_iv: "6afe75b7fe3544429901815523e3fa5c"
server_key_version: "encryptionkey@0"
I20260926 01:13:49.069733 18250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384993574762-392-0/minicluster-data/ts-2/wal/instance:
uuid: "f0035afb0aef4c888bff975c35b48794"
format_stamp: "Formatted at 2026-09-26 01:13:49 on dist-test-slave-7b22"
server_key: "ceda051d092078b891505bf9cfda06c0"
server_key_iv: "6afe75b7fe3544429901815523e3fa5c"
server_key_version: "encryptionkey@0"
I20260926 01:13:49.071017 18250 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 01:13:49.071761 18264 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:13:49.071966 18250 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 01:13:49.072059 18250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384993574762-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384993574762-392-0/minicluster-data/ts-2/wal
uuid: "f0035afb0aef4c888bff975c35b48794"
format_stamp: "Formatted at 2026-09-26 01:13:49 on dist-test-slave-7b22"
server_key: "ceda051d092078b891505bf9cfda06c0"
server_key_iv: "6afe75b7fe3544429901815523e3fa5c"
server_key_version: "encryptionkey@0"
I20260926 01:13:49.072134 18250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384993574762-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384993574762-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384993574762-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:13:49.084926 18250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:13:49.085264 18250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:13:49.085398 18250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:13:49.085649 18250 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:13:49.086048 18250 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:13:49.086087 18250 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:13:49.086120 18250 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:13:49.086143 18250 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:13:49.091635 18250 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38403
I20260926 01:13:49.091681 18377 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38403 every 8 connection(s)
I20260926 01:13:49.092005 18250 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384993574762-392-0/minicluster-data/ts-2/data/info.pb
I20260926 01:13:49.097047 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 18250
I20260926 01:13:49.097158 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384993574762-392-0/minicluster-data/ts-2/wal/instance
I20260926 01:13:49.097340 392 external_mini_cluster.cc:1428] Setting key e4f02f37230a5292bb7a71d3e5f02cea
I20260926 01:13:49.097345 17968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:49.092722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34301 (local address 127.0.98.62:39787)
0926 01:13:49.092808 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0926 01:13:49.092810 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:13:49.093008 (+ 198us) server_negotiation.cc:407] Connection header received
0926 01:13:49.093510 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:13:49.093511 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:13:49.093523 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:13:49.093597 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:13:49.094421 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:49.094966 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:49.096004 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:49.096202 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:49.096267 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:49.097073 (+ 806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:13:49.097075 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:13:49.097075 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:13:49.097081 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:13:49.097096 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:13:49.097100 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:13:49.097144 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0926 01:13:49.097224 (+ 80us) server_negotiation.cc:299] Negotiation successful
0926 01:13:49.097261 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 01:13:49.097604 18380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:49.092662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39787 (local address 127.0.98.3:34301)
0926 01:13:49.092875 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0926 01:13:49.092919 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 01:13:49.093419 (+ 500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:13:49.093718 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:13:49.093722 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:13:49.093731 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:13:49.094311 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:13:49.094316 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:13:49.095074 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:13:49.095075 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:13:49.095892 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:13:49.095895 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:13:49.096272 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:13:49.096273 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:13:49.096383 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:49.096913 (+ 530us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 01:13:49.096914 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:13:49.096942 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 01:13:49.096967 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:13:49.097150 (+ 183us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:13:49.097156 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:13:49.097159 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 01:13:49.097222 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 01:13:49.097342 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":107,"threads_started":1}
I20260926 01:13:49.098035 18378 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39787
I20260926 01:13:49.098143 18378 heartbeater.cc:464] Registering TS with master...
I20260926 01:13:49.098325 18378 heartbeater.cc:511] Master 127.0.98.62:39787 requested a full tablet report, sending...
I20260926 01:13:49.098776 17927 ts_manager.cc:194] Registered new tserver with Master: f0035afb0aef4c888bff975c35b48794 (127.0.98.3:38403)
I20260926 01:13:49.099171 17927 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:34301
I20260926 01:13:49.109694 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 01:13:49.114049 17968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:49.111223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39276 (local address 127.0.98.62:39787)
0926 01:13:49.111301 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0926 01:13:49.111303 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:13:49.111316 (+ 13us) server_negotiation.cc:407] Connection header received
0926 01:13:49.111393 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:13:49.111394 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:13:49.111404 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:13:49.111449 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:13:49.111601 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:49.112094 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:49.112855 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:49.113067 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:49.113122 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:49.113763 (+ 641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:13:49.113764 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:13:49.113765 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:13:49.113770 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:13:49.113787 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:13:49.113790 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:13:49.113877 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 01:13:49.113933 (+ 56us) server_negotiation.cc:299] Negotiation successful
0926 01:13:49.113960 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 01:13:49.115770 17927 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39276:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:13:49.117455 17927 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:13:49.123725 18387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:49.119701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40797 (local address 127.0.0.1:43240)
0926 01:13:49.120037 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0926 01:13:49.120047 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 01:13:49.120094 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:13:49.120494 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:13:49.120495 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:13:49.120497 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:13:49.120531 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:13:49.120532 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:13:49.121792 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:13:49.121793 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:13:49.122629 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:13:49.122632 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:13:49.123400 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:13:49.123402 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:13:49.123487 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:49.123502 (+ 15us) client_negotiation.cc:769] Sending connection context
0926 01:13:49.123577 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 01:13:49.123635 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"spinlock_wait_cycles":896,"thread_start_us":106,"threads_started":1}
I20260926 01:13:49.123735 18388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:49.119802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43240 (local address 127.0.98.2:40797)
0926 01:13:49.120319 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0926 01:13:49.120322 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:13:49.120329 (+ 7us) server_negotiation.cc:407] Connection header received
0926 01:13:49.120366 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:13:49.120368 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:13:49.120382 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:13:49.120463 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:13:49.120610 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:49.121703 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:49.122719 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:49.123314 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:49.123385 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:49.123490 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 01:13:49.123593 (+ 103us) server_negotiation.cc:299] Negotiation successful
0926 01:13:49.123635 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":484,"thread_start_us":75,"threads_started":1}
I20260926 01:13:49.124240 18386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:49.119699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39677 (local address 127.0.0.1:51930)
0926 01:13:49.119834 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0926 01:13:49.119879 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 01:13:49.119944 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:13:49.120288 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:13:49.120291 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:13:49.120294 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:13:49.120371 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:13:49.120373 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:13:49.121705 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:13:49.121706 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:13:49.122823 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:13:49.122826 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:13:49.123988 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:13:49.123989 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:13:49.124052 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:49.124056 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 01:13:49.124106 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 01:13:49.124151 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"mutex_wait_us":2,"thread_start_us":83,"threads_started":1}
I20260926 01:13:49.124256 18389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:49.119888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51930 (local address 127.0.98.1:39677)
0926 01:13:49.120075 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 01:13:49.120079 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 01:13:49.120089 (+ 10us) server_negotiation.cc:407] Connection header received
0926 01:13:49.120122 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:13:49.120124 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:13:49.120140 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:13:49.120213 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:13:49.120449 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:49.121617 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:49.122923 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:49.123782 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:49.123865 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:49.123954 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 01:13:49.124130 (+ 176us) server_negotiation.cc:299] Negotiation successful
0926 01:13:49.124177 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":89,"threads_started":1}
I20260926 01:13:49.124507 18181 tablet_service.cc:1467] Processing CreateTablet for tablet e2127b33aba149809cf27c75eeb151c2 (DEFAULT_TABLE table=test-table [id=6705f5b0baa9433b9d1ec7a59a2b7c0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:13:49.124689 18390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:49.119925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38403 (local address 127.0.0.1:40448)
0926 01:13:49.120180 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0926 01:13:49.120188 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 01:13:49.120222 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:13:49.120371 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:13:49.120372 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:13:49.120374 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:13:49.120425 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:13:49.120427 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:13:49.121448 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:13:49.121450 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:13:49.122836 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:13:49.122838 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:13:49.124511 (+ 1673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:13:49.124512 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:13:49.124558 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:49.124561 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 01:13:49.124592 (+ 31us) client_negotiation.cc:240] Negotiation successful
0926 01:13:49.124625 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"mutex_wait_us":43,"thread_start_us":60,"threads_started":1}
I20260926 01:13:49.124877 18181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2127b33aba149809cf27c75eeb151c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:13:49.124972 18391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:49.120033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40448 (local address 127.0.98.3:38403)
0926 01:13:49.120216 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 01:13:49.120219 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:13:49.120229 (+ 10us) server_negotiation.cc:407] Connection header received
0926 01:13:49.120272 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:13:49.120274 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:13:49.120287 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:13:49.120363 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:13:49.120515 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:49.121370 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:49.122923 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:49.123782 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:49.124191 (+ 409us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:49.124832 (+ 641us) server_negotiation.cc:1035] Waiting for connection context
0926 01:13:49.124856 (+ 24us) server_negotiation.cc:299] Negotiation successful
0926 01:13:49.124894 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":86,"threads_started":1}
I20260926 01:13:49.125566 18312 tablet_service.cc:1467] Processing CreateTablet for tablet e2127b33aba149809cf27c75eeb151c2 (DEFAULT_TABLE table=test-table [id=6705f5b0baa9433b9d1ec7a59a2b7c0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:13:49.125876 18312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2127b33aba149809cf27c75eeb151c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:13:49.127743 18392 tablet_bootstrap.cc:492] T e2127b33aba149809cf27c75eeb151c2 P e26dffe37b4c4797bf680d9a425fd4cd: Bootstrap starting.
I20260926 01:13:49.128396 18393 tablet_bootstrap.cc:492] T e2127b33aba149809cf27c75eeb151c2 P f0035afb0aef4c888bff975c35b48794: Bootstrap starting.
I20260926 01:13:49.128522 18392 tablet_bootstrap.cc:654] T e2127b33aba149809cf27c75eeb151c2 P e26dffe37b4c4797bf680d9a425fd4cd: Neither blocks nor log segments found. Creating new log.
I20260926 01:13:49.128607 18050 tablet_service.cc:1467] Processing CreateTablet for tablet e2127b33aba149809cf27c75eeb151c2 (DEFAULT_TABLE table=test-table [id=6705f5b0baa9433b9d1ec7a59a2b7c0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:13:49.128842 18392 log.cc:824] T e2127b33aba149809cf27c75eeb151c2 P e26dffe37b4c4797bf680d9a425fd4cd: Log is configured to *not* fsync() on all Append() calls
I20260926 01:13:49.128966 18050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2127b33aba149809cf27c75eeb151c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:13:49.129006 18393 tablet_bootstrap.cc:654] T e2127b33aba149809cf27c75eeb151c2 P f0035afb0aef4c888bff975c35b48794: Neither blocks nor log segments found. Creating new log.
I20260926 01:13:49.129253 18393 log.cc:824] T e2127b33aba149809cf27c75eeb151c2 P f0035afb0aef4c888bff975c35b48794: Log is configured to *not* fsync() on all Append() calls
I20260926 01:13:49.129972 18393 tablet_bootstrap.cc:492] T e2127b33aba149809cf27c75eeb151c2 P f0035afb0aef4c888bff975c35b48794: No bootstrap required, opened a new log
I20260926 01:13:49.130036 18393 ts_tablet_manager.cc:1397] T e2127b33aba149809cf27c75eeb151c2 P f0035afb0aef4c888bff975c35b48794: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 01:13:49.130076 18392 tablet_bootstrap.cc:492] T e2127b33aba149809cf27c75eeb151c2 P e26dffe37b4c4797bf680d9a425fd4cd: No bootstrap required, opened a new log
I20260926 01:13:49.130147 18392 ts_tablet_manager.cc:1397] T e2127b33aba149809cf27c75eeb151c2 P e26dffe37b4c4797bf680d9a425fd4cd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 01:13:49.132184 18393 raft_consensus.cc:348] T e2127b33aba149809cf27c75eeb151c2 P f0035afb0aef4c888bff975c35b48794 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26dffe37b4c4797bf680d9a425fd4cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40797 } } peers { permanent_uuid: "56e13026ccd5462f85b8425460b95c6c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39677 } } peers { permanent_uuid: "f0035afb0aef4c888bff975c35b48794" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38403 } }
I20260926 01:13:49.132373 18393 raft_consensus.cc:374] T e2127b33aba149809cf27c75eeb151c2 P f0035afb0aef4c888bff975c35b48794 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:13:49.132411 18393 raft_consensus.cc:729] T e2127b33aba149809cf27c75eeb151c2 P f0035afb0aef4c888bff975c35b48794 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0035afb0aef4c888bff975c35b48794, State: Initialized, Role: FOLLOWER
I20260926 01:13:49.132326 18396 tablet_bootstrap.cc:492] T e2127b33aba149809cf27c75eeb151c2 P 56e13026ccd5462f85b8425460b95c6c: Bootstrap starting.
I20260926 01:13:49.132524 18393 consensus_queue.cc:260] T e2127b33aba149809cf27c75eeb151c2 P f0035afb0aef4c888bff975c35b48794 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26dffe37b4c4797bf680d9a425fd4cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40797 } } peers { permanent_uuid: "56e13026ccd5462f85b8425460b95c6c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39677 } } peers { permanent_uuid: "f0035afb0aef4c888bff975c35b48794" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38403 } }
I20260926 01:13:49.132622 18392 raft_consensus.cc:348] T e2127b33aba149809cf27c75eeb151c2 P e26dffe37b4c4797bf680d9a425fd4cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26dffe37b4c4797bf680d9a425fd4cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40797 } } peers { permanent_uuid: "56e13026ccd5462f85b8425460b95c6c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39677 } } peers { permanent_uuid: "f0035afb0aef4c888bff975c35b48794" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38403 } }
I20260926 01:13:49.132778 18392 raft_consensus.cc:374] T e2127b33aba149809cf27c75eeb151c2 P e26dffe37b4c4797bf680d9a425fd4cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:13:49.132793 18393 ts_tablet_manager.cc:1428] T e2127b33aba149809cf27c75eeb151c2 P f0035afb0aef4c888bff975c35b48794: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 01:13:49.132812 18392 raft_consensus.cc:729] T e2127b33aba149809cf27c75eeb151c2 P e26dffe37b4c4797bf680d9a425fd4cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e26dffe37b4c4797bf680d9a425fd4cd, State: Initialized, Role: FOLLOWER
I20260926 01:13:49.132911 18392 consensus_queue.cc:260] T e2127b33aba149809cf27c75eeb151c2 P e26dffe37b4c4797bf680d9a425fd4cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26dffe37b4c4797bf680d9a425fd4cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40797 } } peers { permanent_uuid: "56e13026ccd5462f85b8425460b95c6c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39677 } } peers { permanent_uuid: "f0035afb0aef4c888bff975c35b48794" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38403 } }
I20260926 01:13:49.133118 18378 heartbeater.cc:503] Master 127.0.98.62:39787 was elected leader, sending a full tablet report...
I20260926 01:13:49.133168 18392 ts_tablet_manager.cc:1428] T e2127b33aba149809cf27c75eeb151c2 P e26dffe37b4c4797bf680d9a425fd4cd: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260926 01:13:49.133186 18396 tablet_bootstrap.cc:654] T e2127b33aba149809cf27c75eeb151c2 P 56e13026ccd5462f85b8425460b95c6c: Neither blocks nor log segments found. Creating new log.
I20260926 01:13:49.133219 18247 heartbeater.cc:503] Master 127.0.98.62:39787 was elected leader, sending a full tablet report...
I20260926 01:13:49.133493 18396 log.cc:824] T e2127b33aba149809cf27c75eeb151c2 P 56e13026ccd5462f85b8425460b95c6c: Log is configured to *not* fsync() on all Append() calls
I20260926 01:13:49.134436 18396 tablet_bootstrap.cc:492] T e2127b33aba149809cf27c75eeb151c2 P 56e13026ccd5462f85b8425460b95c6c: No bootstrap required, opened a new log
I20260926 01:13:49.134506 18396 ts_tablet_manager.cc:1397] T e2127b33aba149809cf27c75eeb151c2 P 56e13026ccd5462f85b8425460b95c6c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 01:13:49.136539 18396 raft_consensus.cc:348] T e2127b33aba149809cf27c75eeb151c2 P 56e13026ccd5462f85b8425460b95c6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26dffe37b4c4797bf680d9a425fd4cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40797 } } peers { permanent_uuid: "56e13026ccd5462f85b8425460b95c6c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39677 } } peers { permanent_uuid: "f0035afb0aef4c888bff975c35b48794" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38403 } }
I20260926 01:13:49.136684 18396 raft_consensus.cc:374] T e2127b33aba149809cf27c75eeb151c2 P 56e13026ccd5462f85b8425460b95c6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:13:49.136705 18396 raft_consensus.cc:729] T e2127b33aba149809cf27c75eeb151c2 P 56e13026ccd5462f85b8425460b95c6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56e13026ccd5462f85b8425460b95c6c, State: Initialized, Role: FOLLOWER
I20260926 01:13:49.136821 18396 consensus_queue.cc:260] T e2127b33aba149809cf27c75eeb151c2 P 56e13026ccd5462f85b8425460b95c6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26dffe37b4c4797bf680d9a425fd4cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40797 } } peers { permanent_uuid: "56e13026ccd5462f85b8425460b95c6c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39677 } } peers { permanent_uuid: "f0035afb0aef4c888bff975c35b48794" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38403 } }
I20260926 01:13:49.137071 18396 ts_tablet_manager.cc:1428] T e2127b33aba149809cf27c75eeb151c2 P 56e13026ccd5462f85b8425460b95c6c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 01:13:49.137166 18116 heartbeater.cc:503] Master 127.0.98.62:39787 was elected leader, sending a full tablet report...
I20260926 01:13:49.204610 18398 raft_consensus.cc:482] T e2127b33aba149809cf27c75eeb151c2 P e26dffe37b4c4797bf680d9a425fd4cd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:13:49.204779 18398 raft_consensus.cc:504] T e2127b33aba149809cf27c75eeb151c2 P e26dffe37b4c4797bf680d9a425fd4cd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26dffe37b4c4797bf680d9a425fd4cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40797 } } peers { permanent_uuid: "56e13026ccd5462f85b8425460b95c6c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39677 } } peers { permanent_uuid: "f0035afb0aef4c888bff975c35b48794" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38403 } }
I20260926 01:13:49.205114 18398 leader_election.cc:288] T e2127b33aba149809cf27c75eeb151c2 P e26dffe37b4c4797bf680d9a425fd4cd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 56e13026ccd5462f85b8425460b95c6c (127.0.98.1:39677), f0035afb0aef4c888bff975c35b48794 (127.0.98.3:38403)
I20260926 01:13:49.208380 18391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:49.205471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47951 (local address 127.0.98.3:38403)
0926 01:13:49.205522 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 01:13:49.205524 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:13:49.205788 (+ 264us) server_negotiation.cc:407] Connection header received
0926 01:13:49.205857 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:13:49.205858 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:13:49.205868 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:13:49.205935 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:13:49.206069 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:49.206615 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:49.207650 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:49.208035 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:49.208091 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:49.208102 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 01:13:49.208259 (+ 157us) server_negotiation.cc:299] Negotiation successful
0926 01:13:49.208307 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 01:13:49.208379 18401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:49.205370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38403 (local address 127.0.98.2:47951)
0926 01:13:49.205606 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0926 01:13:49.205617 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 01:13:49.205762 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:13:49.205941 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:13:49.205942 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:13:49.205944 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:13:49.206009 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:13:49.206011 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:13:49.206716 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:13:49.206717 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:13:49.207537 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:13:49.207540 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:13:49.208119 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:13:49.208120 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:13:49.208188 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:49.208193 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 01:13:49.208242 (+ 49us) client_negotiation.cc:240] Negotiation successful
0926 01:13:49.208282 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"mutex_wait_us":22,"thread_start_us":109,"threads_started":1}
I20260926 01:13:49.208432 18389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:49.205472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40931 (local address 127.0.98.1:39677)
0926 01:13:49.205547 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0926 01:13:49.205549 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:13:49.205559 (+ 10us) server_negotiation.cc:407] Connection header received
0926 01:13:49.205592 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:13:49.205593 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:13:49.205605 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:13:49.205701 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:13:49.205942 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:49.206501 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:49.207678 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:49.208073 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:49.208167 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:49.208178 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 01:13:49.208331 (+ 153us) server_negotiation.cc:299] Negotiation successful
0926 01:13:49.208374 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 01:13:49.208488 18249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:49.205370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39677 (local address 127.0.98.2:40931)
0926 01:13:49.205442 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0926 01:13:49.205455 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 01:13:49.205504 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:13:49.205765 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:13:49.205766 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:13:49.205768 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:13:49.205877 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:13:49.205879 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:13:49.206597 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:13:49.206598 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:13:49.207553 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:13:49.207556 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:13:49.208194 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:13:49.208195 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:13:49.208256 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:49.208260 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 01:13:49.208313 (+ 53us) client_negotiation.cc:240] Negotiation successful
0926 01:13:49.208356 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52,"mutex_wait_us":32}
I20260926 01:13:49.209233 18332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2127b33aba149809cf27c75eeb151c2" candidate_uuid: "e26dffe37b4c4797bf680d9a425fd4cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0035afb0aef4c888bff975c35b48794" is_pre_election: true
I20260926 01:13:49.209317 18070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2127b33aba149809cf27c75eeb151c2" candidate_uuid: "e26dffe37b4c4797bf680d9a425fd4cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56e13026ccd5462f85b8425460b95c6c" is_pre_election: true
I20260926 01:13:49.209401 18332 raft_consensus.cc:2454] T e2127b33aba149809cf27c75eeb151c2 P f0035afb0aef4c888bff975c35b48794 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e26dffe37b4c4797bf680d9a425fd4cd in term 0.
I20260926 01:13:49.209470 18070 raft_consensus.cc:2454] T e2127b33aba149809cf27c75eeb151c2 P 56e13026ccd5462f85b8425460b95c6c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e26dffe37b4c4797bf680d9a425fd4cd in term 0.
I20260926 01:13:49.209636 18136 leader_election.cc:302] T e2127b33aba149809cf27c75eeb151c2 P e26dffe37b4c4797bf680d9a425fd4cd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e26dffe37b4c4797bf680d9a425fd4cd, f0035afb0aef4c888bff975c35b48794; no voters:
I20260926 01:13:49.209846 18398 raft_consensus.cc:2781] T e2127b33aba149809cf27c75eeb151c2 P e26dffe37b4c4797bf680d9a425fd4cd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:13:49.209904 18398 raft_consensus.cc:482] T e2127b33aba149809cf27c75eeb151c2 P e26dffe37b4c4797bf680d9a425fd4cd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:13:49.209926 18398 raft_consensus.cc:3037] T e2127b33aba149809cf27c75eeb151c2 P e26dffe37b4c4797bf680d9a425fd4cd [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:13:49.210726 18398 raft_consensus.cc:504] T e2127b33aba149809cf27c75eeb151c2 P e26dffe37b4c4797bf680d9a425fd4cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26dffe37b4c4797bf680d9a425fd4cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40797 } } peers { permanent_uuid: "56e13026ccd5462f85b8425460b95c6c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39677 } } peers { permanent_uuid: "f0035afb0aef4c888bff975c35b48794" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38403 } }
I20260926 01:13:49.210861 18398 leader_election.cc:288] T e2127b33aba149809cf27c75eeb151c2 P e26dffe37b4c4797bf680d9a425fd4cd [CANDIDATE]: Term 1 election: Requested vote from peers 56e13026ccd5462f85b8425460b95c6c (127.0.98.1:39677), f0035afb0aef4c888bff975c35b48794 (127.0.98.3:38403)
I20260926 01:13:49.211046 18070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2127b33aba149809cf27c75eeb151c2" candidate_uuid: "e26dffe37b4c4797bf680d9a425fd4cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56e13026ccd5462f85b8425460b95c6c"
I20260926 01:13:49.211144 18332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2127b33aba149809cf27c75eeb151c2" candidate_uuid: "e26dffe37b4c4797bf680d9a425fd4cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0035afb0aef4c888bff975c35b48794"
I20260926 01:13:49.211136 18070 raft_consensus.cc:3037] T e2127b33aba149809cf27c75eeb151c2 P 56e13026ccd5462f85b8425460b95c6c [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:13:49.211231 18332 raft_consensus.cc:3037] T e2127b33aba149809cf27c75eeb151c2 P f0035afb0aef4c888bff975c35b48794 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:13:49.211931 18332 raft_consensus.cc:2454] T e2127b33aba149809cf27c75eeb151c2 P f0035afb0aef4c888bff975c35b48794 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e26dffe37b4c4797bf680d9a425fd4cd in term 1.
I20260926 01:13:49.211941 18070 raft_consensus.cc:2454] T e2127b33aba149809cf27c75eeb151c2 P 56e13026ccd5462f85b8425460b95c6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e26dffe37b4c4797bf680d9a425fd4cd in term 1.
I20260926 01:13:49.212086 18136 leader_election.cc:302] T e2127b33aba149809cf27c75eeb151c2 P e26dffe37b4c4797bf680d9a425fd4cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e26dffe37b4c4797bf680d9a425fd4cd, f0035afb0aef4c888bff975c35b48794; no voters:
I20260926 01:13:49.212210 18398 raft_consensus.cc:2781] T e2127b33aba149809cf27c75eeb151c2 P e26dffe37b4c4797bf680d9a425fd4cd [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:13:49.212358 18398 raft_consensus.cc:686] T e2127b33aba149809cf27c75eeb151c2 P e26dffe37b4c4797bf680d9a425fd4cd [term 1 LEADER]: Becoming Leader. State: Replica: e26dffe37b4c4797bf680d9a425fd4cd, State: Running, Role: LEADER
I20260926 01:13:49.212451 18398 consensus_queue.cc:237] T e2127b33aba149809cf27c75eeb151c2 P e26dffe37b4c4797bf680d9a425fd4cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26dffe37b4c4797bf680d9a425fd4cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40797 } } peers { permanent_uuid: "56e13026ccd5462f85b8425460b95c6c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39677 } } peers { permanent_uuid: "f0035afb0aef4c888bff975c35b48794" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38403 } }
I20260926 01:13:49.213192 17928 catalog_manager.cc:5495] T e2127b33aba149809cf27c75eeb151c2 P e26dffe37b4c4797bf680d9a425fd4cd reported cstate change: term changed from 0 to 1, leader changed from <none> to e26dffe37b4c4797bf680d9a425fd4cd (127.0.98.2). New cstate: current_term: 1 leader_uuid: "e26dffe37b4c4797bf680d9a425fd4cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26dffe37b4c4797bf680d9a425fd4cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "56e13026ccd5462f85b8425460b95c6c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39677 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f0035afb0aef4c888bff975c35b48794" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38403 } health_report { overall_health: UNKNOWN } } }
W20260926 01:13:49.218375 18248 tablet.cc:2369] T e2127b33aba149809cf27c75eeb151c2 P e26dffe37b4c4797bf680d9a425fd4cd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 01:13:49.241097 18388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:49.238700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43244 (local address 127.0.98.2:40797)
0926 01:13:49.238754 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 01:13:49.238756 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:13:49.238764 (+ 8us) server_negotiation.cc:407] Connection header received
0926 01:13:49.238806 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:13:49.238807 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:13:49.238824 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:13:49.238886 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:13:49.239134 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:49.239635 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:49.240428 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:49.240633 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:49.240700 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:49.240809 (+ 109us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:13:49.240918 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:13:49.240975 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 01:13:49.240991 (+ 16us) server_negotiation.cc:299] Negotiation successful
0926 01:13:49.241007 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 01:13:49.242967 18070 raft_consensus.cc:1261] T e2127b33aba149809cf27c75eeb151c2 P 56e13026ccd5462f85b8425460b95c6c [term 1 FOLLOWER]: Refusing update from remote peer e26dffe37b4c4797bf680d9a425fd4cd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:13:49.242980 18332 raft_consensus.cc:1261] T e2127b33aba149809cf27c75eeb151c2 P f0035afb0aef4c888bff975c35b48794 [term 1 FOLLOWER]: Refusing update from remote peer e26dffe37b4c4797bf680d9a425fd4cd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:13:49.243296 18398 consensus_queue.cc:1035] T e2127b33aba149809cf27c75eeb151c2 P e26dffe37b4c4797bf680d9a425fd4cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0035afb0aef4c888bff975c35b48794" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:13:49.243436 18402 consensus_queue.cc:1035] T e2127b33aba149809cf27c75eeb151c2 P e26dffe37b4c4797bf680d9a425fd4cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "56e13026ccd5462f85b8425460b95c6c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:13:49.246675 18407 mvcc.cc:205] Tried to move back new op lower bound from 7333417898976550912 to 7333417898854789120. Current Snapshot: MvccSnapshot[applied={T|T < 7333417898976550912}]
I20260926 01:13:49.247253 18406 mvcc.cc:205] Tried to move back new op lower bound from 7333417898976550912 to 7333417898854789120. Current Snapshot: MvccSnapshot[applied={T|T < 7333417898976550912 or (T in {7333417898976550912})}]
I20260926 01:13:49.248549 17927 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39276:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:13:49.248664 17927 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39276:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:13:49.249485 17927 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 21659de43a0f423bb95e88e366c675bd: Sending DeleteTablet for 3 replicas of tablet e2127b33aba149809cf27c75eeb151c2
I20260926 01:13:49.249867 18181 tablet_service.cc:1514] Processing DeleteTablet for tablet e2127b33aba149809cf27c75eeb151c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:13:49 UTC) from {username='slave'} at 127.0.0.1:43240
I20260926 01:13:49.249882 18312 tablet_service.cc:1514] Processing DeleteTablet for tablet e2127b33aba149809cf27c75eeb151c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:13:49 UTC) from {username='slave'} at 127.0.0.1:40448
I20260926 01:13:49.249881 18050 tablet_service.cc:1514] Processing DeleteTablet for tablet e2127b33aba149809cf27c75eeb151c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:13:49 UTC) from {username='slave'} at 127.0.0.1:51930
I20260926 01:13:49.250087 18414 tablet_replica.cc:330] T e2127b33aba149809cf27c75eeb151c2 P e26dffe37b4c4797bf680d9a425fd4cd: stopping tablet replica
I20260926 01:13:49.250175 18414 raft_consensus.cc:2229] T e2127b33aba149809cf27c75eeb151c2 P e26dffe37b4c4797bf680d9a425fd4cd [term 1 LEADER]: Raft consensus shutting down.
I20260926 01:13:49.250272 18414 raft_consensus.cc:2258] T e2127b33aba149809cf27c75eeb151c2 P e26dffe37b4c4797bf680d9a425fd4cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:13:49.250381 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 17988
I20260926 01:13:49.252830 18414 ts_tablet_manager.cc:1905] T e2127b33aba149809cf27c75eeb151c2 P e26dffe37b4c4797bf680d9a425fd4cd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:13:49.254599 18414 ts_tablet_manager.cc:1918] T e2127b33aba149809cf27c75eeb151c2 P e26dffe37b4c4797bf680d9a425fd4cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:13:49.254694 18414 log.cc:1196] T e2127b33aba149809cf27c75eeb151c2 P e26dffe37b4c4797bf680d9a425fd4cd: Deleting WAL directory at /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384993574762-392-0/minicluster-data/ts-1/wal/wals/e2127b33aba149809cf27c75eeb151c2
I20260926 01:13:49.254936 18414 ts_tablet_manager.cc:1939] T e2127b33aba149809cf27c75eeb151c2 P e26dffe37b4c4797bf680d9a425fd4cd: Deleting consensus metadata
I20260926 01:13:49.255786 17914 catalog_manager.cc:4841] TS e26dffe37b4c4797bf680d9a425fd4cd (127.0.98.2:40797): tablet e2127b33aba149809cf27c75eeb151c2 (table test-table [id=6705f5b0baa9433b9d1ec7a59a2b7c0e]) successfully deleted
W20260926 01:13:49.257272 17912 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39677 (error 108)
I20260926 01:13:49.257423 18415 tablet_replica.cc:330] T e2127b33aba149809cf27c75eeb151c2 P f0035afb0aef4c888bff975c35b48794: stopping tablet replica
I20260926 01:13:49.257562 18415 raft_consensus.cc:2229] T e2127b33aba149809cf27c75eeb151c2 P f0035afb0aef4c888bff975c35b48794 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:13:49.257719 18415 raft_consensus.cc:2258] T e2127b33aba149809cf27c75eeb151c2 P f0035afb0aef4c888bff975c35b48794 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:13:49.257891 18390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:49.257708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39677 (local address 127.0.0.1:51936)
0926 01:13:49.257767 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0926 01:13:49.257851 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260926 01:13:49.257964 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 18119
W20260926 01:13:49.258038 17912 catalog_manager.cc:4584] TS 56e13026ccd5462f85b8425460b95c6c (127.0.98.1:39677): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e2127b33aba149809cf27c75eeb151c2: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39677: connect: Connection refused (error 111)
I20260926 01:13:49.262697 18415 ts_tablet_manager.cc:1905] T e2127b33aba149809cf27c75eeb151c2 P f0035afb0aef4c888bff975c35b48794: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:13:49.263044 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 18250
I20260926 01:13:49.267627 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 17897
I20260926 01:13:49.267792 18390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:49.267597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38403 (local address 127.0.0.1:40450)
0926 01:13:49.267662 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0926 01:13:49.267696 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.3:38403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
2026-09-26T01:13:49Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (763 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384993574762-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 01:13:49 dist-test-slave-7b22 krb5kdc[18420](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 01:13:49 dist-test-slave-7b22 krb5kdc[18420](info): set up 2 sockets
Sep 26 01:13:49 dist-test-slave-7b22 krb5kdc[18420](info): commencing operation
krb5kdc: starting...
W20260926 01:13:50.225443 392 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384993574762-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384993574762-392-0/krb5kdc/test-user.keytab.
Sep 26 01:13:50 dist-test-slave-7b22 krb5kdc[18420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385230, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T01:13:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:13:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384993574762-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384993574762-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 01:13:50.342396 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384993574762-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384993574762-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384993574762-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37011
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384993574762-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37011
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:13:50.422838 18433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:13:50.423061 18433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:13:50.423094 18433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:13:50.424273 18433 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:13:50.424319 18433 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:13:50.424335 18433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:13:50.424351 18433 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:13:50.424365 18433 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:13:50.425889 18433 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384993574762-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384993574762-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37011
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384993574762-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37011
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18433
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384993574762-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:13:50.426133 18433 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:13:50.426350 18433 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 01:13:50.431354 18433 server_base.cc:1034] running on GCE node
W20260926 01:13:50.431337 18438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:13:50.431346 18439 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:13:50.431337 18441 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:13:50.431866 18433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:13:50.432155 18433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:13:50.433301 18433 hybrid_clock.cc:648] HybridClock initialized: now 1790385230433288 us; error 32 us; skew 500 ppm
Sep 26 01:13:50 dist-test-slave-7b22 krb5kdc[18420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385230, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:13:50.435787 18433 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 01:13:50.436239 18433 webserver.cc:460] Webserver started at http://127.0.98.62:45003/ using document root <none> and password file <none>
I20260926 01:13:50.436456 18433 fs_manager.cc:359] Metadata directory not provided
I20260926 01:13:50.436504 18433 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:13:50.436628 18433 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:13:50.437661 18433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384993574762-392-0/minicluster-data/master-0/data/instance:
uuid: "b47fe7b952554bd5b1960dd1d66241eb"
format_stamp: "Formatted at 2026-09-26 01:13:50 on dist-test-slave-7b22"
server_key: "7d97f0991ad63077704d59b43b373ff5"
server_key_iv: "7e121bcbca5212ff88c0521b1463bc42"
server_key_version: "encryptionkey@0"
I20260926 01:13:50.438020 18433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384993574762-392-0/minicluster-data/master-0/wal/instance:
uuid: "b47fe7b952554bd5b1960dd1d66241eb"
format_stamp: "Formatted at 2026-09-26 01:13:50 on dist-test-slave-7b22"
server_key: "7d97f0991ad63077704d59b43b373ff5"
server_key_iv: "7e121bcbca5212ff88c0521b1463bc42"
server_key_version: "encryptionkey@0"
I20260926 01:13:50.439366 18433 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:13:50.440078 18448 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:13:50.440233 18433 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 01:13:50.440308 18433 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384993574762-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384993574762-392-0/minicluster-data/master-0/wal
uuid: "b47fe7b952554bd5b1960dd1d66241eb"
format_stamp: "Formatted at 2026-09-26 01:13:50 on dist-test-slave-7b22"
server_key: "7d97f0991ad63077704d59b43b373ff5"
server_key_iv: "7e121bcbca5212ff88c0521b1463bc42"
server_key_version: "encryptionkey@0"
I20260926 01:13:50.440371 18433 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384993574762-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384993574762-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384993574762-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:13:50.462949 18433 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:13:50.465078 18433 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:13:50.465237 18433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:13:50.469275 18433 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37011
I20260926 01:13:50.469321 18500 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37011 every 8 connection(s)
I20260926 01:13:50.469655 18433 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
I20260926 01:13:50.470507 18501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:13:50.472925 18501 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b47fe7b952554bd5b1960dd1d66241eb: Bootstrap starting.
I20260926 01:13:50.473520 18501 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b47fe7b952554bd5b1960dd1d66241eb: Neither blocks nor log segments found. Creating new log.
I20260926 01:13:50.473790 18501 log.cc:824] T 00000000000000000000000000000000 P b47fe7b952554bd5b1960dd1d66241eb: Log is configured to *not* fsync() on all Append() calls
I20260926 01:13:50.474550 18501 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b47fe7b952554bd5b1960dd1d66241eb: No bootstrap required, opened a new log
I20260926 01:13:50.476239 18501 raft_consensus.cc:348] T 00000000000000000000000000000000 P b47fe7b952554bd5b1960dd1d66241eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b47fe7b952554bd5b1960dd1d66241eb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37011 } }
I20260926 01:13:50.476397 18501 raft_consensus.cc:374] T 00000000000000000000000000000000 P b47fe7b952554bd5b1960dd1d66241eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:13:50.476418 18501 raft_consensus.cc:729] T 00000000000000000000000000000000 P b47fe7b952554bd5b1960dd1d66241eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b47fe7b952554bd5b1960dd1d66241eb, State: Initialized, Role: FOLLOWER
I20260926 01:13:50.476555 18501 consensus_queue.cc:260] T 00000000000000000000000000000000 P b47fe7b952554bd5b1960dd1d66241eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b47fe7b952554bd5b1960dd1d66241eb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37011 } }
I20260926 01:13:50.476629 18501 raft_consensus.cc:388] T 00000000000000000000000000000000 P b47fe7b952554bd5b1960dd1d66241eb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:13:50.476656 18501 raft_consensus.cc:482] T 00000000000000000000000000000000 P b47fe7b952554bd5b1960dd1d66241eb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:13:50.476706 18501 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b47fe7b952554bd5b1960dd1d66241eb [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:13:50.477293 18501 raft_consensus.cc:504] T 00000000000000000000000000000000 P b47fe7b952554bd5b1960dd1d66241eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b47fe7b952554bd5b1960dd1d66241eb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37011 } }
I20260926 01:13:50.477387 18501 leader_election.cc:302] T 00000000000000000000000000000000 P b47fe7b952554bd5b1960dd1d66241eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b47fe7b952554bd5b1960dd1d66241eb; no voters:
I20260926 01:13:50.477555 18501 leader_election.cc:288] T 00000000000000000000000000000000 P b47fe7b952554bd5b1960dd1d66241eb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:13:50.477644 18504 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b47fe7b952554bd5b1960dd1d66241eb [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:13:50.477764 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 18433
I20260926 01:13:50.477840 18501 sys_catalog.cc:564] T 00000000000000000000000000000000 P b47fe7b952554bd5b1960dd1d66241eb [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:13:50.477893 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384993574762-392-0/minicluster-data/master-0/wal/instance
I20260926 01:13:50.477900 18504 raft_consensus.cc:686] T 00000000000000000000000000000000 P b47fe7b952554bd5b1960dd1d66241eb [term 1 LEADER]: Becoming Leader. State: Replica: b47fe7b952554bd5b1960dd1d66241eb, State: Running, Role: LEADER
I20260926 01:13:50.477989 18504 consensus_queue.cc:237] T 00000000000000000000000000000000 P b47fe7b952554bd5b1960dd1d66241eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b47fe7b952554bd5b1960dd1d66241eb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37011 } }
I20260926 01:13:50.478097 392 external_mini_cluster.cc:1428] Setting key 57bddab330fc1a5d5a67739e111d15df
I20260926 01:13:50.478879 18506 sys_catalog.cc:455] T 00000000000000000000000000000000 P b47fe7b952554bd5b1960dd1d66241eb [sys.catalog]: SysCatalogTable state changed. Reason: New leader b47fe7b952554bd5b1960dd1d66241eb. Latest consensus state: current_term: 1 leader_uuid: "b47fe7b952554bd5b1960dd1d66241eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b47fe7b952554bd5b1960dd1d66241eb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37011 } } }
I20260926 01:13:50.478885 18505 sys_catalog.cc:455] T 00000000000000000000000000000000 P b47fe7b952554bd5b1960dd1d66241eb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b47fe7b952554bd5b1960dd1d66241eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b47fe7b952554bd5b1960dd1d66241eb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37011 } } }
I20260926 01:13:50.479040 18505 sys_catalog.cc:458] T 00000000000000000000000000000000 P b47fe7b952554bd5b1960dd1d66241eb [sys.catalog]: This master's current role is: LEADER
I20260926 01:13:50.479228 18506 sys_catalog.cc:458] T 00000000000000000000000000000000 P b47fe7b952554bd5b1960dd1d66241eb [sys.catalog]: This master's current role is: LEADER
I20260926 01:13:50.479264 18519 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:13:50.479854 18519 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:13:50.481467 18519 catalog_manager.cc:1347] Generated new cluster ID: 277964c1c63c4396b734f93f956edbf6
I20260926 01:13:50.481525 18519 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 01:13:50 dist-test-slave-7b22 krb5kdc[18420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385230, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:13:50.489517 18514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:50.478608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53684 (local address 127.0.98.62:37011)
0926 01:13:50.478803 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 01:13:50.478813 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 01:13:50.479604 (+ 791us) server_negotiation.cc:407] Connection header received
0926 01:13:50.479746 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:13:50.479752 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:13:50.479800 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:13:50.479931 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:13:50.480854 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:50.482025 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:50.482810 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:50.483266 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:50.483361 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:50.485684 (+ 2323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:13:50.485690 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:13:50.485690 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:13:50.485705 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:13:50.487968 (+ 2263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:13:50.488326 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:13:50.488330 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:13:50.488331 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:13:50.488439 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:13:50.488621 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:13:50.488622 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:13:50.488622 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:13:50.488878 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:13:50.489043 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0926 01:13:50.489144 (+ 101us) server_negotiation.cc:299] Negotiation successful
0926 01:13:50.489294 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":87,"threads_started":1}
I20260926 01:13:50.509636 18519 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:13:50.510363 18519 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:13:50.514146 18519 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b47fe7b952554bd5b1960dd1d66241eb: Generated new TSK 0
I20260926 01:13:50.514465 18519 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384993574762-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384993574762-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 01:13:50.544625 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384993574762-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384993574762-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384993574762-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384993574762-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37011
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384993574762-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:13:50.641281 18527 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:13:50.641520 18527 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:13:50.641547 18527 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:13:50.642869 18527 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:13:50.642923 18527 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:13:50.642953 18527 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 01:13:50.644587 18527 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384993574762-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384993574762-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384993574762-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384993574762-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37011
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18527
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384993574762-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:13:50.644841 18527 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:13:50.645071 18527 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:13:50.650352 18532 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:13:50.650384 18533 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:13:50.650388 18535 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:13:50.650692 18527 server_base.cc:1034] running on GCE node
I20260926 01:13:50.650928 18527 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:13:50.651188 18527 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:13:50.652346 18527 hybrid_clock.cc:648] HybridClock initialized: now 1790385230652333 us; error 33 us; skew 500 ppm
Sep 26 01:13:50 dist-test-slave-7b22 krb5kdc[18420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385230, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:13:50.655000 18527 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 01:13:50.655462 18527 webserver.cc:460] Webserver started at http://127.0.98.1:40893/ using document root <none> and password file <none>
I20260926 01:13:50.655702 18527 fs_manager.cc:359] Metadata directory not provided
I20260926 01:13:50.655754 18527 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:13:50.655889 18527 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:13:50.656978 18527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384993574762-392-0/minicluster-data/ts-0/data/instance:
uuid: "20d1514e15ca47aa9ef7a51f561f5315"
format_stamp: "Formatted at 2026-09-26 01:13:50 on dist-test-slave-7b22"
server_key: "480f4fc23ff2a1be601bd69318e4d706"
server_key_iv: "dd88cfb12b2e6a4ad8da5c975a72c11e"
server_key_version: "encryptionkey@0"
I20260926 01:13:50.657312 18527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384993574762-392-0/minicluster-data/ts-0/wal/instance:
uuid: "20d1514e15ca47aa9ef7a51f561f5315"
format_stamp: "Formatted at 2026-09-26 01:13:50 on dist-test-slave-7b22"
server_key: "480f4fc23ff2a1be601bd69318e4d706"
server_key_iv: "dd88cfb12b2e6a4ad8da5c975a72c11e"
server_key_version: "encryptionkey@0"
I20260926 01:13:50.658732 18527 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260926 01:13:50.659533 18542 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:13:50.659857 18527 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:13:50.659965 18527 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384993574762-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384993574762-392-0/minicluster-data/ts-0/wal
uuid: "20d1514e15ca47aa9ef7a51f561f5315"
format_stamp: "Formatted at 2026-09-26 01:13:50 on dist-test-slave-7b22"
server_key: "480f4fc23ff2a1be601bd69318e4d706"
server_key_iv: "dd88cfb12b2e6a4ad8da5c975a72c11e"
server_key_version: "encryptionkey@0"
I20260926 01:13:50.660037 18527 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384993574762-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384993574762-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384993574762-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:13:50.677701 18527 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:13:50.680248 18527 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:13:50.680393 18527 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:13:50.680683 18527 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:13:50.680989 18527 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:13:50.681022 18527 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:13:50.681053 18527 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:13:50.681070 18527 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:13:50.686758 18527 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43433
I20260926 01:13:50.686815 18655 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43433 every 8 connection(s)
I20260926 01:13:50.687121 18527 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384993574762-392-0/minicluster-data/ts-0/data/info.pb
I20260926 01:13:50.690789 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 18527
I20260926 01:13:50.690909 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384993574762-392-0/minicluster-data/ts-0/wal/instance
I20260926 01:13:50.691089 392 external_mini_cluster.cc:1428] Setting key 622565e815d88b944a31fcb932cefd2c
Sep 26 01:13:50 dist-test-slave-7b22 krb5kdc[18420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385230, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:13:50.698097 18514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:50.687831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47003 (local address 127.0.98.62:37011)
0926 01:13:50.687904 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0926 01:13:50.687906 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:13:50.688172 (+ 266us) server_negotiation.cc:407] Connection header received
0926 01:13:50.689233 (+ 1061us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:13:50.689234 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:13:50.689245 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:13:50.689304 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:13:50.690072 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:50.690599 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:50.691907 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:50.692136 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:50.692194 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:50.694820 (+ 2626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:13:50.694822 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:13:50.694822 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:13:50.694830 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:13:50.696639 (+ 1809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:13:50.697083 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:13:50.697085 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:13:50.697085 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:13:50.697146 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:13:50.697402 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:13:50.697403 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:13:50.697403 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:13:50.697525 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:13:50.697584 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 01:13:50.697897 (+ 313us) server_negotiation.cc:299] Negotiation successful
0926 01:13:50.698003 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 01:13:50.698282 18658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:50.687803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37011 (local address 127.0.98.1:47003)
0926 01:13:50.688020 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0926 01:13:50.688069 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 01:13:50.689051 (+ 982us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:13:50.689378 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:13:50.689382 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:13:50.689718 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:13:50.689987 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:13:50.689993 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:13:50.690710 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:13:50.690711 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:13:50.691817 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:13:50.691821 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:13:50.692223 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:13:50.692224 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:13:50.692373 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:50.692957 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:13:50.692967 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:13:50.694703 (+ 1736us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:13:50.696742 (+ 2039us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:13:50.696753 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:13:50.696756 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:13:50.697010 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:13:50.697213 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:13:50.697214 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:13:50.697214 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:13:50.697335 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:13:50.697585 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:13:50.697592 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:13:50.697747 (+ 155us) client_negotiation.cc:769] Sending connection context
0926 01:13:50.697863 (+ 116us) client_negotiation.cc:240] Negotiation successful
0926 01:13:50.698040 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":79,"threads_started":1}
I20260926 01:13:50.698817 18656 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37011
I20260926 01:13:50.698912 18656 heartbeater.cc:464] Registering TS with master...
I20260926 01:13:50.699092 18656 heartbeater.cc:511] Master 127.0.98.62:37011 requested a full tablet report, sending...
I20260926 01:13:50.699621 18465 ts_manager.cc:194] Registered new tserver with Master: 20d1514e15ca47aa9ef7a51f561f5315 (127.0.98.1:43433)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260926 01:13:50.700271 18465 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:47003
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384993574762-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384993574762-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 01:13:50.721386 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384993574762-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384993574762-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384993574762-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384993574762-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37011
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384993574762-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:13:50.803457 18661 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:13:50.803696 18661 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:13:50.803731 18661 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:13:50.804939 18661 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:13:50.804981 18661 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:13:50.805009 18661 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 01:13:50.806602 18661 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384993574762-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384993574762-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384993574762-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384993574762-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37011
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18661
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384993574762-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:13:50.806828 18661 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:13:50.807017 18661 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 01:13:50.812381 18661 server_base.cc:1034] running on GCE node
W20260926 01:13:50.812361 18669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:13:50.812361 18666 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:13:50.812341 18667 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:13:50.812776 18661 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:13:50.813051 18661 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:13:50.814306 18661 hybrid_clock.cc:648] HybridClock initialized: now 1790385230814263 us; error 34 us; skew 500 ppm
Sep 26 01:13:50 dist-test-slave-7b22 krb5kdc[18420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385230, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:13:50.817377 18661 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 01:13:50.817885 18661 webserver.cc:460] Webserver started at http://127.0.98.2:39073/ using document root <none> and password file <none>
I20260926 01:13:50.818114 18661 fs_manager.cc:359] Metadata directory not provided
I20260926 01:13:50.818178 18661 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:13:50.818307 18661 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:13:50.819409 18661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384993574762-392-0/minicluster-data/ts-1/data/instance:
uuid: "5b8615ce8610482e8e4b7d7fa9fd9494"
format_stamp: "Formatted at 2026-09-26 01:13:50 on dist-test-slave-7b22"
server_key: "6dabea490b1e1cefaf1463ac19b4a4ac"
server_key_iv: "4b003e413680ffe1eb27f26a7eca1b7a"
server_key_version: "encryptionkey@0"
I20260926 01:13:50.819751 18661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384993574762-392-0/minicluster-data/ts-1/wal/instance:
uuid: "5b8615ce8610482e8e4b7d7fa9fd9494"
format_stamp: "Formatted at 2026-09-26 01:13:50 on dist-test-slave-7b22"
server_key: "6dabea490b1e1cefaf1463ac19b4a4ac"
server_key_iv: "4b003e413680ffe1eb27f26a7eca1b7a"
server_key_version: "encryptionkey@0"
I20260926 01:13:50.821085 18661 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 01:13:50.821928 18676 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:13:50.822110 18661 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 01:13:50.822208 18661 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384993574762-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384993574762-392-0/minicluster-data/ts-1/wal
uuid: "5b8615ce8610482e8e4b7d7fa9fd9494"
format_stamp: "Formatted at 2026-09-26 01:13:50 on dist-test-slave-7b22"
server_key: "6dabea490b1e1cefaf1463ac19b4a4ac"
server_key_iv: "4b003e413680ffe1eb27f26a7eca1b7a"
server_key_version: "encryptionkey@0"
I20260926 01:13:50.822274 18661 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384993574762-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384993574762-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384993574762-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:13:50.838704 18661 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:13:50.840982 18661 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:13:50.841130 18661 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:13:50.841365 18661 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:13:50.841676 18661 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:13:50.841710 18661 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:13:50.841741 18661 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:13:50.841758 18661 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:13:50.847591 18661 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33023
I20260926 01:13:50.847653 18789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33023 every 8 connection(s)
I20260926 01:13:50.847954 18661 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384993574762-392-0/minicluster-data/ts-1/data/info.pb
Sep 26 01:13:50 dist-test-slave-7b22 krb5kdc[18420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385230, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:13:50.856674 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 18661
I20260926 01:13:50.856778 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384993574762-392-0/minicluster-data/ts-1/wal/instance
I20260926 01:13:50.856980 392 external_mini_cluster.cc:1428] Setting key 4781c063213436c5853e4986339e8e86
I20260926 01:13:50.857363 18514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:50.848708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57711 (local address 127.0.98.62:37011)
0926 01:13:50.848789 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0926 01:13:50.848791 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:13:50.848946 (+ 155us) server_negotiation.cc:407] Connection header received
0926 01:13:50.849920 (+ 974us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:13:50.849921 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:13:50.849935 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:13:50.849995 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:13:50.850669 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:50.851171 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:50.852200 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:50.852393 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:50.852451 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:50.854577 (+ 2126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:13:50.854579 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:13:50.854579 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:13:50.854588 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:13:50.855860 (+ 1272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:13:50.856268 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:13:50.856269 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:13:50.856270 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:13:50.856324 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:13:50.856590 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:13:50.856592 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:13:50.856592 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:13:50.856729 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:13:50.856804 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 01:13:50.857188 (+ 384us) server_negotiation.cc:299] Negotiation successful
0926 01:13:50.857285 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 01:13:50.857620 18792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:50.848669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37011 (local address 127.0.98.2:57711)
0926 01:13:50.848827 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 01:13:50.848867 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 01:13:50.849789 (+ 922us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:13:50.850068 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:13:50.850071 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:13:50.850373 (+ 302us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:13:50.850571 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:13:50.850575 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:13:50.851262 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:13:50.851264 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:13:50.852105 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:13:50.852107 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:13:50.852466 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:13:50.852466 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:13:50.852587 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:50.853087 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:13:50.853095 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:13:50.854483 (+ 1388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:13:50.855953 (+ 1470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:13:50.855961 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:13:50.855964 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:13:50.856190 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:13:50.856380 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:13:50.856380 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:13:50.856381 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:13:50.856508 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:13:50.856802 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:13:50.856809 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:13:50.856987 (+ 178us) client_negotiation.cc:769] Sending connection context
0926 01:13:50.857141 (+ 154us) client_negotiation.cc:240] Negotiation successful
0926 01:13:50.857336 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":65,"threads_started":1}
I20260926 01:13:50.858186 18790 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37011
I20260926 01:13:50.858286 18790 heartbeater.cc:464] Registering TS with master...
I20260926 01:13:50.858469 18790 heartbeater.cc:511] Master 127.0.98.62:37011 requested a full tablet report, sending...
I20260926 01:13:50.858822 18465 ts_manager.cc:194] Registered new tserver with Master: 5b8615ce8610482e8e4b7d7fa9fd9494 (127.0.98.2:33023)
I20260926 01:13:50.859318 18465 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:57711
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384993574762-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384993574762-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 01:13:50.882540 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384993574762-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384993574762-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384993574762-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384993574762-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37011
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384993574762-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:13:50.965080 18795 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:13:50.965307 18795 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:13:50.965332 18795 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:13:50.966661 18795 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:13:50.966712 18795 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:13:50.966744 18795 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 01:13:50.968343 18795 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384993574762-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384993574762-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384993574762-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384993574762-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37011
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18795
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384993574762-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:13:50.968556 18795 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:13:50.968747 18795 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 01:13:50.973990 18795 server_base.cc:1034] running on GCE node
W20260926 01:13:50.973977 18803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:13:50.973958 18800 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:13:50.973949 18801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:13:50.974474 18795 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:13:50.974722 18795 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:13:50.975875 18795 hybrid_clock.cc:648] HybridClock initialized: now 1790385230975864 us; error 33 us; skew 500 ppm
Sep 26 01:13:50 dist-test-slave-7b22 krb5kdc[18420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385230, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:13:50.978746 18795 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 01:13:50.979185 18795 webserver.cc:460] Webserver started at http://127.0.98.3:45081/ using document root <none> and password file <none>
I20260926 01:13:50.979426 18795 fs_manager.cc:359] Metadata directory not provided
I20260926 01:13:50.979475 18795 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:13:50.979614 18795 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:13:50.980702 18795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384993574762-392-0/minicluster-data/ts-2/data/instance:
uuid: "9626075dbd3444e0923912055ca6f299"
format_stamp: "Formatted at 2026-09-26 01:13:50 on dist-test-slave-7b22"
server_key: "b4eb140cb83a66125cef511a098454c1"
server_key_iv: "6535ec956e7a391448f3f6394bdd8624"
server_key_version: "encryptionkey@0"
I20260926 01:13:50.981035 18795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384993574762-392-0/minicluster-data/ts-2/wal/instance:
uuid: "9626075dbd3444e0923912055ca6f299"
format_stamp: "Formatted at 2026-09-26 01:13:50 on dist-test-slave-7b22"
server_key: "b4eb140cb83a66125cef511a098454c1"
server_key_iv: "6535ec956e7a391448f3f6394bdd8624"
server_key_version: "encryptionkey@0"
I20260926 01:13:50.982429 18795 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 01:13:50.983232 18810 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:13:50.983387 18795 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 01:13:50.983469 18795 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384993574762-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384993574762-392-0/minicluster-data/ts-2/wal
uuid: "9626075dbd3444e0923912055ca6f299"
format_stamp: "Formatted at 2026-09-26 01:13:50 on dist-test-slave-7b22"
server_key: "b4eb140cb83a66125cef511a098454c1"
server_key_iv: "6535ec956e7a391448f3f6394bdd8624"
server_key_version: "encryptionkey@0"
I20260926 01:13:50.983533 18795 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384993574762-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384993574762-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384993574762-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:13:50.995530 18795 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:13:50.997713 18795 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:13:50.997951 18795 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:13:50.998181 18795 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:13:50.998484 18795 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:13:50.998523 18795 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:13:50.998559 18795 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:13:50.998579 18795 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:13:51.004920 18795 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37597
I20260926 01:13:51.004988 18923 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37597 every 8 connection(s)
I20260926 01:13:51.005297 18795 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384993574762-392-0/minicluster-data/ts-2/data/info.pb
I20260926 01:13:51.007490 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 18795
I20260926 01:13:51.007599 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384993574762-392-0/minicluster-data/ts-2/wal/instance
I20260926 01:13:51.007773 392 external_mini_cluster.cc:1428] Setting key 9ec13e2692104c3876c57b3023ae7eeb
Sep 26 01:13:51 dist-test-slave-7b22 krb5kdc[18420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385230, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:13:51.015766 18514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:51.006105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39619 (local address 127.0.98.62:37011)
0926 01:13:51.006171 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 01:13:51.006173 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:13:51.006375 (+ 202us) server_negotiation.cc:407] Connection header received
0926 01:13:51.007350 (+ 975us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:13:51.007351 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:13:51.007362 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:13:51.007404 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:13:51.008200 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:51.008823 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:51.010046 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:51.010261 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:51.010333 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:51.012774 (+ 2441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:13:51.012776 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:13:51.012776 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:13:51.012784 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:13:51.014355 (+ 1571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:13:51.014771 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:13:51.014773 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:13:51.014773 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:13:51.014849 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:13:51.015088 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:13:51.015089 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:13:51.015089 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:13:51.015229 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:13:51.015298 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 01:13:51.015616 (+ 318us) server_negotiation.cc:299] Negotiation successful
0926 01:13:51.015692 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 01:13:51.015889 18926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:51.006070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37011 (local address 127.0.98.3:39619)
0926 01:13:51.006254 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0926 01:13:51.006293 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 01:13:51.007236 (+ 943us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:13:51.007480 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:13:51.007485 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:13:51.007842 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:13:51.008111 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:13:51.008118 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:13:51.008951 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:13:51.008952 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:13:51.009968 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:13:51.009971 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:13:51.010344 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:13:51.010344 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:13:51.010455 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:51.011043 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:13:51.011052 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:13:51.012673 (+ 1621us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:13:51.014432 (+ 1759us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:13:51.014439 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:13:51.014441 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:13:51.014670 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:13:51.014907 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:13:51.014907 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:13:51.014908 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:13:51.015012 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:13:51.015306 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:13:51.015311 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:13:51.015447 (+ 136us) client_negotiation.cc:769] Sending connection context
0926 01:13:51.015541 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 01:13:51.015671 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":83,"threads_started":1}
I20260926 01:13:51.016323 18924 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37011
I20260926 01:13:51.016415 18924 heartbeater.cc:464] Registering TS with master...
I20260926 01:13:51.016559 18924 heartbeater.cc:511] Master 127.0.98.62:37011 requested a full tablet report, sending...
I20260926 01:13:51.016916 18465 ts_manager.cc:194] Registered new tserver with Master: 9626075dbd3444e0923912055ca6f299 (127.0.98.3:37597)
I20260926 01:13:51.017277 18465 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:39619
I20260926 01:13:51.019443 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 01:13:51.026909 18514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:51.020686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41264 (local address 127.0.98.62:37011)
0926 01:13:51.020750 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 01:13:51.020751 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 01:13:51.020856 (+ 105us) server_negotiation.cc:407] Connection header received
0926 01:13:51.020890 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:13:51.020891 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:13:51.020899 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:13:51.020937 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:13:51.021469 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:51.022005 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:51.022750 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:51.022934 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:51.022987 (+ 53us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:51.024342 (+ 1355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:13:51.024343 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:13:51.024344 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:13:51.024350 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:13:51.025584 (+ 1234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:13:51.026126 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:13:51.026128 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:13:51.026128 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:13:51.026207 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:13:51.026447 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:13:51.026448 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:13:51.026448 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:13:51.026603 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:13:51.026684 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 01:13:51.026784 (+ 100us) server_negotiation.cc:299] Negotiation successful
0926 01:13:51.026838 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 01:13:51.028554 18465 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41264:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:13:51.030167 18465 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:13:51.036995 18937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:51.032785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41390 (local address 127.0.98.2:33023)
0926 01:13:51.032945 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0926 01:13:51.032950 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 01:13:51.033401 (+ 451us) server_negotiation.cc:407] Connection header received
0926 01:13:51.033466 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:13:51.033468 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:13:51.033483 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:13:51.033548 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:13:51.033770 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:51.034893 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:51.035835 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:51.036577 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:51.036657 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:51.036724 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 01:13:51.036862 (+ 138us) server_negotiation.cc:299] Negotiation successful
0926 01:13:51.036911 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":83,"threads_started":1}
I20260926 01:13:51.037037 18936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:51.032651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33023 (local address 127.0.0.1:41390)
0926 01:13:51.033326 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0926 01:13:51.033332 (+ 6us) client_negotiation.cc:174] Beginning negotiation
0926 01:13:51.033365 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:13:51.033666 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:13:51.033667 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:13:51.033669 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:13:51.033700 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:13:51.033703 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:13:51.034991 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:13:51.034992 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:13:51.035748 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:13:51.035750 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:13:51.036664 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:13:51.036665 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:13:51.036746 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:51.036761 (+ 15us) client_negotiation.cc:769] Sending connection context
0926 01:13:51.036839 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 01:13:51.036897 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":659,"thread_start_us":45,"threads_started":1}
I20260926 01:13:51.037232 18939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:51.032950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52146 (local address 127.0.98.1:43433)
0926 01:13:51.033112 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0926 01:13:51.033116 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 01:13:51.033506 (+ 390us) server_negotiation.cc:407] Connection header received
0926 01:13:51.033553 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:13:51.033555 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:13:51.033570 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:13:51.033647 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:13:51.033875 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:51.034555 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:51.035930 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:51.036632 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:51.036689 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:51.036736 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0926 01:13:51.037062 (+ 326us) server_negotiation.cc:299] Negotiation successful
0926 01:13:51.037159 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":74,"threads_started":1}
I20260926 01:13:51.037340 18935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:51.032827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43433 (local address 127.0.0.1:52146)
0926 01:13:51.033404 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0926 01:13:51.033409 (+ 5us) client_negotiation.cc:174] Beginning negotiation
0926 01:13:51.033444 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:13:51.033744 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:13:51.033744 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:13:51.033746 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:13:51.033791 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:13:51.033792 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:13:51.034653 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:13:51.034654 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:13:51.035846 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:13:51.035849 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:13:51.036770 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:13:51.036771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:13:51.036835 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:51.036842 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 01:13:51.036889 (+ 47us) client_negotiation.cc:240] Negotiation successful
0926 01:13:51.036930 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"mutex_wait_us":4,"thread_start_us":65,"threads_started":1}
I20260926 01:13:51.037876 18938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:51.032648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37597 (local address 127.0.0.1:54042)
0926 01:13:51.033141 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0926 01:13:51.033174 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 01:13:51.033237 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:13:51.033507 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:13:51.033510 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:13:51.033512 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:13:51.033585 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:13:51.033587 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:13:51.034967 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:13:51.034967 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:13:51.036273 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:13:51.036275 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:13:51.037433 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:13:51.037434 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:13:51.037664 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:51.037670 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 01:13:51.037732 (+ 62us) client_negotiation.cc:240] Negotiation successful
0926 01:13:51.037772 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":82,"threads_started":1}
I20260926 01:13:51.037923 18940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:51.032958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54042 (local address 127.0.98.3:37597)
0926 01:13:51.033135 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 01:13:51.033139 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 01:13:51.033222 (+ 83us) server_negotiation.cc:407] Connection header received
0926 01:13:51.033311 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:13:51.033312 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:13:51.033325 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:13:51.033401 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:13:51.033665 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:51.034859 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:51.036354 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:51.036933 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:51.036996 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:51.037048 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 01:13:51.037768 (+ 720us) server_negotiation.cc:299] Negotiation successful
0926 01:13:51.037843 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":78,"threads_started":1}
I20260926 01:13:51.037915 18724 tablet_service.cc:1467] Processing CreateTablet for tablet 9b2c077704604ccf9c54c87efe11142a (DEFAULT_TABLE table=test-table [id=665d2386a7fc421e9298a46ce301cfca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:13:51.038056 18590 tablet_service.cc:1467] Processing CreateTablet for tablet 9b2c077704604ccf9c54c87efe11142a (DEFAULT_TABLE table=test-table [id=665d2386a7fc421e9298a46ce301cfca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:13:51.038300 18724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b2c077704604ccf9c54c87efe11142a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:13:51.038396 18590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b2c077704604ccf9c54c87efe11142a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:13:51.038832 18858 tablet_service.cc:1467] Processing CreateTablet for tablet 9b2c077704604ccf9c54c87efe11142a (DEFAULT_TABLE table=test-table [id=665d2386a7fc421e9298a46ce301cfca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:13:51.039125 18858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b2c077704604ccf9c54c87efe11142a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:13:51.041183 18941 tablet_bootstrap.cc:492] T 9b2c077704604ccf9c54c87efe11142a P 5b8615ce8610482e8e4b7d7fa9fd9494: Bootstrap starting.
I20260926 01:13:51.042011 18942 tablet_bootstrap.cc:492] T 9b2c077704604ccf9c54c87efe11142a P 20d1514e15ca47aa9ef7a51f561f5315: Bootstrap starting.
I20260926 01:13:51.042084 18941 tablet_bootstrap.cc:654] T 9b2c077704604ccf9c54c87efe11142a P 5b8615ce8610482e8e4b7d7fa9fd9494: Neither blocks nor log segments found. Creating new log.
I20260926 01:13:51.042634 18941 log.cc:824] T 9b2c077704604ccf9c54c87efe11142a P 5b8615ce8610482e8e4b7d7fa9fd9494: Log is configured to *not* fsync() on all Append() calls
I20260926 01:13:51.042695 18942 tablet_bootstrap.cc:654] T 9b2c077704604ccf9c54c87efe11142a P 20d1514e15ca47aa9ef7a51f561f5315: Neither blocks nor log segments found. Creating new log.
I20260926 01:13:51.043009 18943 tablet_bootstrap.cc:492] T 9b2c077704604ccf9c54c87efe11142a P 9626075dbd3444e0923912055ca6f299: Bootstrap starting.
I20260926 01:13:51.043241 18942 log.cc:824] T 9b2c077704604ccf9c54c87efe11142a P 20d1514e15ca47aa9ef7a51f561f5315: Log is configured to *not* fsync() on all Append() calls
I20260926 01:13:51.043442 18941 tablet_bootstrap.cc:492] T 9b2c077704604ccf9c54c87efe11142a P 5b8615ce8610482e8e4b7d7fa9fd9494: No bootstrap required, opened a new log
I20260926 01:13:51.043505 18941 ts_tablet_manager.cc:1397] T 9b2c077704604ccf9c54c87efe11142a P 5b8615ce8610482e8e4b7d7fa9fd9494: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 01:13:51.043632 18943 tablet_bootstrap.cc:654] T 9b2c077704604ccf9c54c87efe11142a P 9626075dbd3444e0923912055ca6f299: Neither blocks nor log segments found. Creating new log.
I20260926 01:13:51.043995 18942 tablet_bootstrap.cc:492] T 9b2c077704604ccf9c54c87efe11142a P 20d1514e15ca47aa9ef7a51f561f5315: No bootstrap required, opened a new log
I20260926 01:13:51.044049 18942 ts_tablet_manager.cc:1397] T 9b2c077704604ccf9c54c87efe11142a P 20d1514e15ca47aa9ef7a51f561f5315: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 01:13:51.044116 18943 log.cc:824] T 9b2c077704604ccf9c54c87efe11142a P 9626075dbd3444e0923912055ca6f299: Log is configured to *not* fsync() on all Append() calls
I20260926 01:13:51.044853 18943 tablet_bootstrap.cc:492] T 9b2c077704604ccf9c54c87efe11142a P 9626075dbd3444e0923912055ca6f299: No bootstrap required, opened a new log
I20260926 01:13:51.044899 18943 ts_tablet_manager.cc:1397] T 9b2c077704604ccf9c54c87efe11142a P 9626075dbd3444e0923912055ca6f299: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 01:13:51.045495 18941 raft_consensus.cc:348] T 9b2c077704604ccf9c54c87efe11142a P 5b8615ce8610482e8e4b7d7fa9fd9494 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b8615ce8610482e8e4b7d7fa9fd9494" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33023 } } peers { permanent_uuid: "9626075dbd3444e0923912055ca6f299" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37597 } } peers { permanent_uuid: "20d1514e15ca47aa9ef7a51f561f5315" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43433 } }
I20260926 01:13:51.045683 18941 raft_consensus.cc:374] T 9b2c077704604ccf9c54c87efe11142a P 5b8615ce8610482e8e4b7d7fa9fd9494 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:13:51.045718 18941 raft_consensus.cc:729] T 9b2c077704604ccf9c54c87efe11142a P 5b8615ce8610482e8e4b7d7fa9fd9494 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b8615ce8610482e8e4b7d7fa9fd9494, State: Initialized, Role: FOLLOWER
I20260926 01:13:51.045871 18941 consensus_queue.cc:260] T 9b2c077704604ccf9c54c87efe11142a P 5b8615ce8610482e8e4b7d7fa9fd9494 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b8615ce8610482e8e4b7d7fa9fd9494" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33023 } } peers { permanent_uuid: "9626075dbd3444e0923912055ca6f299" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37597 } } peers { permanent_uuid: "20d1514e15ca47aa9ef7a51f561f5315" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43433 } }
I20260926 01:13:51.046052 18942 raft_consensus.cc:348] T 9b2c077704604ccf9c54c87efe11142a P 20d1514e15ca47aa9ef7a51f561f5315 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b8615ce8610482e8e4b7d7fa9fd9494" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33023 } } peers { permanent_uuid: "9626075dbd3444e0923912055ca6f299" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37597 } } peers { permanent_uuid: "20d1514e15ca47aa9ef7a51f561f5315" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43433 } }
I20260926 01:13:51.046203 18942 raft_consensus.cc:374] T 9b2c077704604ccf9c54c87efe11142a P 20d1514e15ca47aa9ef7a51f561f5315 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:13:51.046182 18790 heartbeater.cc:503] Master 127.0.98.62:37011 was elected leader, sending a full tablet report...
I20260926 01:13:51.046169 18941 ts_tablet_manager.cc:1428] T 9b2c077704604ccf9c54c87efe11142a P 5b8615ce8610482e8e4b7d7fa9fd9494: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 01:13:51.046235 18942 raft_consensus.cc:729] T 9b2c077704604ccf9c54c87efe11142a P 20d1514e15ca47aa9ef7a51f561f5315 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20d1514e15ca47aa9ef7a51f561f5315, State: Initialized, Role: FOLLOWER
I20260926 01:13:51.046342 18942 consensus_queue.cc:260] T 9b2c077704604ccf9c54c87efe11142a P 20d1514e15ca47aa9ef7a51f561f5315 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b8615ce8610482e8e4b7d7fa9fd9494" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33023 } } peers { permanent_uuid: "9626075dbd3444e0923912055ca6f299" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37597 } } peers { permanent_uuid: "20d1514e15ca47aa9ef7a51f561f5315" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43433 } }
I20260926 01:13:51.046403 18943 raft_consensus.cc:348] T 9b2c077704604ccf9c54c87efe11142a P 9626075dbd3444e0923912055ca6f299 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b8615ce8610482e8e4b7d7fa9fd9494" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33023 } } peers { permanent_uuid: "9626075dbd3444e0923912055ca6f299" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37597 } } peers { permanent_uuid: "20d1514e15ca47aa9ef7a51f561f5315" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43433 } }
I20260926 01:13:51.046594 18942 ts_tablet_manager.cc:1428] T 9b2c077704604ccf9c54c87efe11142a P 20d1514e15ca47aa9ef7a51f561f5315: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 01:13:51.046614 18656 heartbeater.cc:503] Master 127.0.98.62:37011 was elected leader, sending a full tablet report...
I20260926 01:13:51.046556 18943 raft_consensus.cc:374] T 9b2c077704604ccf9c54c87efe11142a P 9626075dbd3444e0923912055ca6f299 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:13:51.046862 18943 raft_consensus.cc:729] T 9b2c077704604ccf9c54c87efe11142a P 9626075dbd3444e0923912055ca6f299 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9626075dbd3444e0923912055ca6f299, State: Initialized, Role: FOLLOWER
I20260926 01:13:51.047127 18943 consensus_queue.cc:260] T 9b2c077704604ccf9c54c87efe11142a P 9626075dbd3444e0923912055ca6f299 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b8615ce8610482e8e4b7d7fa9fd9494" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33023 } } peers { permanent_uuid: "9626075dbd3444e0923912055ca6f299" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37597 } } peers { permanent_uuid: "20d1514e15ca47aa9ef7a51f561f5315" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43433 } }
I20260926 01:13:51.047574 18943 ts_tablet_manager.cc:1428] T 9b2c077704604ccf9c54c87efe11142a P 9626075dbd3444e0923912055ca6f299: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260926 01:13:51.047658 18924 heartbeater.cc:503] Master 127.0.98.62:37011 was elected leader, sending a full tablet report...
W20260926 01:13:51.098532 18791 tablet.cc:2369] T 9b2c077704604ccf9c54c87efe11142a P 5b8615ce8610482e8e4b7d7fa9fd9494: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 01:13:51.176810 18949 raft_consensus.cc:482] T 9b2c077704604ccf9c54c87efe11142a P 9626075dbd3444e0923912055ca6f299 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:13:51.176965 18949 raft_consensus.cc:504] T 9b2c077704604ccf9c54c87efe11142a P 9626075dbd3444e0923912055ca6f299 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b8615ce8610482e8e4b7d7fa9fd9494" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33023 } } peers { permanent_uuid: "9626075dbd3444e0923912055ca6f299" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37597 } } peers { permanent_uuid: "20d1514e15ca47aa9ef7a51f561f5315" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43433 } }
I20260926 01:13:51.177332 18949 leader_election.cc:288] T 9b2c077704604ccf9c54c87efe11142a P 9626075dbd3444e0923912055ca6f299 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b8615ce8610482e8e4b7d7fa9fd9494 (127.0.98.2:33023), 20d1514e15ca47aa9ef7a51f561f5315 (127.0.98.1:43433)
I20260926 01:13:51.180363 18937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:51.177657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53237 (local address 127.0.98.2:33023)
0926 01:13:51.177702 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 01:13:51.177704 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:13:51.177713 (+ 9us) server_negotiation.cc:407] Connection header received
0926 01:13:51.177740 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:13:51.177741 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:13:51.177751 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:13:51.177797 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:13:51.178003 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:51.178553 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:51.179628 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:51.180050 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:51.180113 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:51.180125 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0926 01:13:51.180257 (+ 132us) server_negotiation.cc:299] Negotiation successful
0926 01:13:51.180298 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 01:13:51.180363 18926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:51.177540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33023 (local address 127.0.98.3:53237)
0926 01:13:51.177588 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0926 01:13:51.177624 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 01:13:51.177671 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:13:51.177838 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:13:51.177839 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:13:51.177841 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:13:51.177908 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:13:51.177909 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:13:51.178676 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:13:51.178677 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:13:51.179542 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:13:51.179545 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:13:51.180124 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:13:51.180124 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:13:51.180197 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:51.180199 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 01:13:51.180243 (+ 44us) client_negotiation.cc:240] Negotiation successful
0926 01:13:51.180285 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260926 01:13:51.180437 18939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:51.177619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42071 (local address 127.0.98.1:43433)
0926 01:13:51.177709 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0926 01:13:51.177711 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:13:51.177967 (+ 256us) server_negotiation.cc:407] Connection header received
0926 01:13:51.177998 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:13:51.177998 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:13:51.178011 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:13:51.178070 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:13:51.178228 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:51.178692 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:51.179739 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:51.180107 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:51.180167 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:51.180177 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 01:13:51.180327 (+ 150us) server_negotiation.cc:299] Negotiation successful
0926 01:13:51.180368 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 01:13:51.180657 18950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:51.177543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43433 (local address 127.0.98.3:42071)
0926 01:13:51.177873 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0926 01:13:51.177882 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 01:13:51.177943 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:13:51.178096 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:13:51.178096 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:13:51.178098 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:13:51.178164 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:13:51.178166 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:13:51.178770 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:13:51.178772 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:13:51.179658 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:13:51.179661 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:13:51.180183 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:13:51.180184 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:13:51.180265 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:51.180268 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 01:13:51.180312 (+ 44us) client_negotiation.cc:240] Negotiation successful
0926 01:13:51.180353 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"mutex_wait_us":130,"thread_start_us":108,"threads_started":1}
I20260926 01:13:51.180756 18744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b2c077704604ccf9c54c87efe11142a" candidate_uuid: "9626075dbd3444e0923912055ca6f299" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b8615ce8610482e8e4b7d7fa9fd9494" is_pre_election: true
I20260926 01:13:51.180925 18744 raft_consensus.cc:2454] T 9b2c077704604ccf9c54c87efe11142a P 5b8615ce8610482e8e4b7d7fa9fd9494 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9626075dbd3444e0923912055ca6f299 in term 0.
I20260926 01:13:51.180985 18610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b2c077704604ccf9c54c87efe11142a" candidate_uuid: "9626075dbd3444e0923912055ca6f299" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20d1514e15ca47aa9ef7a51f561f5315" is_pre_election: true
I20260926 01:13:51.181149 18610 raft_consensus.cc:2454] T 9b2c077704604ccf9c54c87efe11142a P 20d1514e15ca47aa9ef7a51f561f5315 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9626075dbd3444e0923912055ca6f299 in term 0.
I20260926 01:13:51.181159 18813 leader_election.cc:302] T 9b2c077704604ccf9c54c87efe11142a P 9626075dbd3444e0923912055ca6f299 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b8615ce8610482e8e4b7d7fa9fd9494, 9626075dbd3444e0923912055ca6f299; no voters:
I20260926 01:13:51.181288 18949 raft_consensus.cc:2781] T 9b2c077704604ccf9c54c87efe11142a P 9626075dbd3444e0923912055ca6f299 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:13:51.181357 18949 raft_consensus.cc:482] T 9b2c077704604ccf9c54c87efe11142a P 9626075dbd3444e0923912055ca6f299 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:13:51.181382 18949 raft_consensus.cc:3037] T 9b2c077704604ccf9c54c87efe11142a P 9626075dbd3444e0923912055ca6f299 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:13:51.182111 18949 raft_consensus.cc:504] T 9b2c077704604ccf9c54c87efe11142a P 9626075dbd3444e0923912055ca6f299 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b8615ce8610482e8e4b7d7fa9fd9494" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33023 } } peers { permanent_uuid: "9626075dbd3444e0923912055ca6f299" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37597 } } peers { permanent_uuid: "20d1514e15ca47aa9ef7a51f561f5315" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43433 } }
I20260926 01:13:51.182243 18949 leader_election.cc:288] T 9b2c077704604ccf9c54c87efe11142a P 9626075dbd3444e0923912055ca6f299 [CANDIDATE]: Term 1 election: Requested vote from peers 5b8615ce8610482e8e4b7d7fa9fd9494 (127.0.98.2:33023), 20d1514e15ca47aa9ef7a51f561f5315 (127.0.98.1:43433)
I20260926 01:13:51.182397 18610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b2c077704604ccf9c54c87efe11142a" candidate_uuid: "9626075dbd3444e0923912055ca6f299" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20d1514e15ca47aa9ef7a51f561f5315"
I20260926 01:13:51.182461 18744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b2c077704604ccf9c54c87efe11142a" candidate_uuid: "9626075dbd3444e0923912055ca6f299" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b8615ce8610482e8e4b7d7fa9fd9494"
I20260926 01:13:51.182489 18610 raft_consensus.cc:3037] T 9b2c077704604ccf9c54c87efe11142a P 20d1514e15ca47aa9ef7a51f561f5315 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:13:51.182545 18744 raft_consensus.cc:3037] T 9b2c077704604ccf9c54c87efe11142a P 5b8615ce8610482e8e4b7d7fa9fd9494 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:13:51.183333 18744 raft_consensus.cc:2454] T 9b2c077704604ccf9c54c87efe11142a P 5b8615ce8610482e8e4b7d7fa9fd9494 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9626075dbd3444e0923912055ca6f299 in term 1.
I20260926 01:13:51.183378 18610 raft_consensus.cc:2454] T 9b2c077704604ccf9c54c87efe11142a P 20d1514e15ca47aa9ef7a51f561f5315 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9626075dbd3444e0923912055ca6f299 in term 1.
I20260926 01:13:51.183499 18813 leader_election.cc:302] T 9b2c077704604ccf9c54c87efe11142a P 9626075dbd3444e0923912055ca6f299 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b8615ce8610482e8e4b7d7fa9fd9494, 9626075dbd3444e0923912055ca6f299; no voters:
I20260926 01:13:51.183590 18949 raft_consensus.cc:2781] T 9b2c077704604ccf9c54c87efe11142a P 9626075dbd3444e0923912055ca6f299 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:13:51.183741 18949 raft_consensus.cc:686] T 9b2c077704604ccf9c54c87efe11142a P 9626075dbd3444e0923912055ca6f299 [term 1 LEADER]: Becoming Leader. State: Replica: 9626075dbd3444e0923912055ca6f299, State: Running, Role: LEADER
I20260926 01:13:51.183825 18949 consensus_queue.cc:237] T 9b2c077704604ccf9c54c87efe11142a P 9626075dbd3444e0923912055ca6f299 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b8615ce8610482e8e4b7d7fa9fd9494" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33023 } } peers { permanent_uuid: "9626075dbd3444e0923912055ca6f299" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37597 } } peers { permanent_uuid: "20d1514e15ca47aa9ef7a51f561f5315" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43433 } }
I20260926 01:13:51.184667 18465 catalog_manager.cc:5495] T 9b2c077704604ccf9c54c87efe11142a P 9626075dbd3444e0923912055ca6f299 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9626075dbd3444e0923912055ca6f299 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "9626075dbd3444e0923912055ca6f299" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b8615ce8610482e8e4b7d7fa9fd9494" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33023 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9626075dbd3444e0923912055ca6f299" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37597 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20d1514e15ca47aa9ef7a51f561f5315" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43433 } health_report { overall_health: UNKNOWN } } }
W20260926 01:13:51.187811 18657 tablet.cc:2369] T 9b2c077704604ccf9c54c87efe11142a P 20d1514e15ca47aa9ef7a51f561f5315: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 01:13:51.223515 18940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:51.221262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54048 (local address 127.0.98.3:37597)
0926 01:13:51.221330 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 01:13:51.221331 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 01:13:51.221338 (+ 7us) server_negotiation.cc:407] Connection header received
0926 01:13:51.221372 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:13:51.221373 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:13:51.221390 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:13:51.221448 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:13:51.221555 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:51.222073 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:51.222794 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:51.222977 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:51.223031 (+ 54us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:51.223180 (+ 149us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:13:51.223271 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:13:51.223359 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 01:13:51.223406 (+ 47us) server_negotiation.cc:299] Negotiation successful
0926 01:13:51.223414 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 01:13:51.225420 18610 raft_consensus.cc:1261] T 9b2c077704604ccf9c54c87efe11142a P 20d1514e15ca47aa9ef7a51f561f5315 [term 1 FOLLOWER]: Refusing update from remote peer 9626075dbd3444e0923912055ca6f299: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:13:51.225400 18744 raft_consensus.cc:1261] T 9b2c077704604ccf9c54c87efe11142a P 5b8615ce8610482e8e4b7d7fa9fd9494 [term 1 FOLLOWER]: Refusing update from remote peer 9626075dbd3444e0923912055ca6f299: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:13:51.225729 18949 consensus_queue.cc:1035] T 9b2c077704604ccf9c54c87efe11142a P 9626075dbd3444e0923912055ca6f299 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20d1514e15ca47aa9ef7a51f561f5315" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:13:51.225890 18951 consensus_queue.cc:1035] T 9b2c077704604ccf9c54c87efe11142a P 9626075dbd3444e0923912055ca6f299 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8615ce8610482e8e4b7d7fa9fd9494" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:13:51.228121 18954 mvcc.cc:205] Tried to move back new op lower bound from 7333417907096412160 to 7333417906929741824. Current Snapshot: MvccSnapshot[applied={T|T < 7333417907096412160}]
I20260926 01:13:51.228890 18956 mvcc.cc:205] Tried to move back new op lower bound from 7333417907096412160 to 7333417906929741824. Current Snapshot: MvccSnapshot[applied={T|T < 7333417907096412160}]
I20260926 01:13:51.231058 18463 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41264:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:13:51.231148 18463 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41264:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:13:51.231889 18463 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b47fe7b952554bd5b1960dd1d66241eb: Sending DeleteTablet for 3 replicas of tablet 9b2c077704604ccf9c54c87efe11142a
I20260926 01:13:51.232292 18590 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b2c077704604ccf9c54c87efe11142a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:13:51 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52146
I20260926 01:13:51.232301 18724 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b2c077704604ccf9c54c87efe11142a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:13:51 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41390
I20260926 01:13:51.232321 18858 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b2c077704604ccf9c54c87efe11142a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:13:51 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54042
I20260926 01:13:51.232518 18965 tablet_replica.cc:330] T 9b2c077704604ccf9c54c87efe11142a P 5b8615ce8610482e8e4b7d7fa9fd9494: stopping tablet replica
I20260926 01:13:51.232537 18963 tablet_replica.cc:330] T 9b2c077704604ccf9c54c87efe11142a P 20d1514e15ca47aa9ef7a51f561f5315: stopping tablet replica
I20260926 01:13:51.232628 18963 raft_consensus.cc:2229] T 9b2c077704604ccf9c54c87efe11142a P 20d1514e15ca47aa9ef7a51f561f5315 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:13:51.232641 18965 raft_consensus.cc:2229] T 9b2c077704604ccf9c54c87efe11142a P 5b8615ce8610482e8e4b7d7fa9fd9494 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:13:51.232733 18963 raft_consensus.cc:2258] T 9b2c077704604ccf9c54c87efe11142a P 20d1514e15ca47aa9ef7a51f561f5315 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:13:51.232827 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 18527
I20260926 01:13:51.232874 18964 tablet_replica.cc:330] T 9b2c077704604ccf9c54c87efe11142a P 9626075dbd3444e0923912055ca6f299: stopping tablet replica
I20260926 01:13:51.232949 18964 raft_consensus.cc:2229] T 9b2c077704604ccf9c54c87efe11142a P 9626075dbd3444e0923912055ca6f299 [term 1 LEADER]: Raft consensus shutting down.
I20260926 01:13:51.233045 18964 raft_consensus.cc:2258] T 9b2c077704604ccf9c54c87efe11142a P 9626075dbd3444e0923912055ca6f299 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:13:51.232872 18965 raft_consensus.cc:2258] T 9b2c077704604ccf9c54c87efe11142a P 5b8615ce8610482e8e4b7d7fa9fd9494 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:13:51.233783 18965 ts_tablet_manager.cc:1905] T 9b2c077704604ccf9c54c87efe11142a P 5b8615ce8610482e8e4b7d7fa9fd9494: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:13:51.233974 18964 ts_tablet_manager.cc:1905] T 9b2c077704604ccf9c54c87efe11142a P 9626075dbd3444e0923912055ca6f299: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:13:51.235668 18965 ts_tablet_manager.cc:1918] T 9b2c077704604ccf9c54c87efe11142a P 5b8615ce8610482e8e4b7d7fa9fd9494: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:13:51.235678 18964 ts_tablet_manager.cc:1918] T 9b2c077704604ccf9c54c87efe11142a P 9626075dbd3444e0923912055ca6f299: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:13:51.235737 18965 log.cc:1196] T 9b2c077704604ccf9c54c87efe11142a P 5b8615ce8610482e8e4b7d7fa9fd9494: Deleting WAL directory at /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384993574762-392-0/minicluster-data/ts-1/wal/wals/9b2c077704604ccf9c54c87efe11142a
I20260926 01:13:51.235747 18964 log.cc:1196] T 9b2c077704604ccf9c54c87efe11142a P 9626075dbd3444e0923912055ca6f299: Deleting WAL directory at /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384993574762-392-0/minicluster-data/ts-2/wal/wals/9b2c077704604ccf9c54c87efe11142a
I20260926 01:13:51.236006 18964 ts_tablet_manager.cc:1939] T 9b2c077704604ccf9c54c87efe11142a P 9626075dbd3444e0923912055ca6f299: Deleting consensus metadata
I20260926 01:13:51.236025 18965 ts_tablet_manager.cc:1939] T 9b2c077704604ccf9c54c87efe11142a P 5b8615ce8610482e8e4b7d7fa9fd9494: Deleting consensus metadata
I20260926 01:13:51.236485 18451 catalog_manager.cc:4841] TS 5b8615ce8610482e8e4b7d7fa9fd9494 (127.0.98.2:33023): tablet 9b2c077704604ccf9c54c87efe11142a (table test-table [id=665d2386a7fc421e9298a46ce301cfca]) successfully deleted
I20260926 01:13:51.236609 18452 catalog_manager.cc:4841] TS 9626075dbd3444e0923912055ca6f299 (127.0.98.3:37597): tablet 9b2c077704604ccf9c54c87efe11142a (table test-table [id=665d2386a7fc421e9298a46ce301cfca]) successfully deleted
W20260926 01:13:51.239647 18452 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43433 (error 108)
I20260926 01:13:51.239858 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 18661
I20260926 01:13:51.242321 18936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:51.242155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43433 (local address 127.0.0.1:52152)
0926 01:13:51.242213 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0926 01:13:51.242276 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260926 01:13:51.242527 18452 catalog_manager.cc:4584] TS 20d1514e15ca47aa9ef7a51f561f5315 (127.0.98.1:43433): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9b2c077704604ccf9c54c87efe11142a: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43433: connect: Connection refused (error 111)
I20260926 01:13:51.244881 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 18795
I20260926 01:13:51.249377 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 18433
2026-09-26T01:13:51Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (1982 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-26T01:13:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:13:51Z Disabled control of system clock
I20260926 01:13:51.275892 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384993574762-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384993574762-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384993574762-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40793
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:39289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40793
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 01:13:51.356143 18970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:13:51.356395 18970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:13:51.356415 18970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:13:51.357664 18970 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:13:51.357721 18970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:13:51.357733 18970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:13:51.357744 18970 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:13:51.357755 18970 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:13:51.359230 18970 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384993574762-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384993574762-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40793
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40793
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18970
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384993574762-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:13:51.359477 18970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:13:51.359792 18970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:13:51.364578 18975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:13:51.364586 18976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:13:51.364787 18978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:13:51.364734 18970 server_base.cc:1034] running on GCE node
I20260926 01:13:51.365145 18970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:13:51.365414 18970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:13:51.366590 18970 hybrid_clock.cc:648] HybridClock initialized: now 1790385231366569 us; error 35 us; skew 500 ppm
I20260926 01:13:51.367987 18970 webserver.cc:460] Webserver started at http://127.0.98.62:37499/ using document root <none> and password file <none>
I20260926 01:13:51.368221 18970 fs_manager.cc:359] Metadata directory not provided
I20260926 01:13:51.368268 18970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:13:51.368379 18970 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:13:51.369494 18970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384993574762-392-0/minicluster-data/master-0/data/instance:
uuid: "4935712ba089437f8cf2abbed64d693f"
format_stamp: "Formatted at 2026-09-26 01:13:51 on dist-test-slave-7b22"
server_key: "39f5ecf84be171f0e86c572cf8d322c3"
server_key_iv: "4431bc1ab186e283c7d383c856f0e394"
server_key_version: "encryptionkey@0"
I20260926 01:13:51.369855 18970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384993574762-392-0/minicluster-data/master-0/wal/instance:
uuid: "4935712ba089437f8cf2abbed64d693f"
format_stamp: "Formatted at 2026-09-26 01:13:51 on dist-test-slave-7b22"
server_key: "39f5ecf84be171f0e86c572cf8d322c3"
server_key_iv: "4431bc1ab186e283c7d383c856f0e394"
server_key_version: "encryptionkey@0"
I20260926 01:13:51.371181 18970 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 01:13:51.371939 18984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:13:51.372102 18970 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 01:13:51.372180 18970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384993574762-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384993574762-392-0/minicluster-data/master-0/wal
uuid: "4935712ba089437f8cf2abbed64d693f"
format_stamp: "Formatted at 2026-09-26 01:13:51 on dist-test-slave-7b22"
server_key: "39f5ecf84be171f0e86c572cf8d322c3"
server_key_iv: "4431bc1ab186e283c7d383c856f0e394"
server_key_version: "encryptionkey@0"
I20260926 01:13:51.372252 18970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384993574762-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384993574762-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384993574762-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:13:51.394798 18970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:13:51.395117 18970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:13:51.395243 18970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:13:51.398904 18970 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40793
I20260926 01:13:51.398952 19036 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40793 every 8 connection(s)
I20260926 01:13:51.399279 18970 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
I20260926 01:13:51.400341 19037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:13:51.400331 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 18970
I20260926 01:13:51.400432 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384993574762-392-0/minicluster-data/master-0/wal/instance
I20260926 01:13:51.400635 392 external_mini_cluster.cc:1428] Setting key 13dfc6d261cb5bdac2467d06d2f908e9
I20260926 01:13:51.403105 19037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4935712ba089437f8cf2abbed64d693f: Bootstrap starting.
I20260926 01:13:51.403708 19037 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4935712ba089437f8cf2abbed64d693f: Neither blocks nor log segments found. Creating new log.
I20260926 01:13:51.404006 19037 log.cc:824] T 00000000000000000000000000000000 P 4935712ba089437f8cf2abbed64d693f: Log is configured to *not* fsync() on all Append() calls
I20260926 01:13:51.404734 19037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4935712ba089437f8cf2abbed64d693f: No bootstrap required, opened a new log
I20260926 01:13:51.406489 19040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:51.401181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52216 (local address 127.0.98.62:40793)
0926 01:13:51.401374 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 01:13:51.401383 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 01:13:51.401630 (+ 247us) server_negotiation.cc:407] Connection header received
0926 01:13:51.402116 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:13:51.402123 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:13:51.402187 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:13:51.402302 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:13:51.402552 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:51.403504 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:51.404491 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:51.404964 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:51.405060 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:51.405855 (+ 795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:13:51.405860 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:13:51.405860 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:13:51.406027 (+ 167us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:13:51.406058 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:13:51.406069 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:13:51.406121 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 01:13:51.406213 (+ 92us) server_negotiation.cc:299] Negotiation successful
0926 01:13:51.406281 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":65,"threads_started":1}
I20260926 01:13:51.406936 19037 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4935712ba089437f8cf2abbed64d693f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4935712ba089437f8cf2abbed64d693f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40793 } }
I20260926 01:13:51.407078 19037 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4935712ba089437f8cf2abbed64d693f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:13:51.407114 19037 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4935712ba089437f8cf2abbed64d693f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4935712ba089437f8cf2abbed64d693f, State: Initialized, Role: FOLLOWER
I20260926 01:13:51.407256 19037 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4935712ba089437f8cf2abbed64d693f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4935712ba089437f8cf2abbed64d693f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40793 } }
I20260926 01:13:51.407330 19037 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4935712ba089437f8cf2abbed64d693f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:13:51.407351 19037 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4935712ba089437f8cf2abbed64d693f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:13:51.407382 19037 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4935712ba089437f8cf2abbed64d693f [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:13:51.407907 19037 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4935712ba089437f8cf2abbed64d693f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4935712ba089437f8cf2abbed64d693f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40793 } }
I20260926 01:13:51.407989 19037 leader_election.cc:302] T 00000000000000000000000000000000 P 4935712ba089437f8cf2abbed64d693f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4935712ba089437f8cf2abbed64d693f; no voters:
I20260926 01:13:51.408129 19037 leader_election.cc:288] T 00000000000000000000000000000000 P 4935712ba089437f8cf2abbed64d693f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:13:51.408182 19042 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4935712ba089437f8cf2abbed64d693f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:13:51.408357 19042 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4935712ba089437f8cf2abbed64d693f [term 1 LEADER]: Becoming Leader. State: Replica: 4935712ba089437f8cf2abbed64d693f, State: Running, Role: LEADER
I20260926 01:13:51.408367 19037 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4935712ba089437f8cf2abbed64d693f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:13:51.408492 19042 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4935712ba089437f8cf2abbed64d693f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4935712ba089437f8cf2abbed64d693f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40793 } }
I20260926 01:13:51.408850 19043 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4935712ba089437f8cf2abbed64d693f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4935712ba089437f8cf2abbed64d693f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4935712ba089437f8cf2abbed64d693f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40793 } } }
I20260926 01:13:51.408972 19043 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4935712ba089437f8cf2abbed64d693f [sys.catalog]: This master's current role is: LEADER
I20260926 01:13:51.408849 19044 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4935712ba089437f8cf2abbed64d693f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4935712ba089437f8cf2abbed64d693f. Latest consensus state: current_term: 1 leader_uuid: "4935712ba089437f8cf2abbed64d693f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4935712ba089437f8cf2abbed64d693f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40793 } } }
I20260926 01:13:51.409191 19044 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4935712ba089437f8cf2abbed64d693f [sys.catalog]: This master's current role is: LEADER
I20260926 01:13:51.409193 19051 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:13:51.409811 19051 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:13:51.411358 19051 catalog_manager.cc:1347] Generated new cluster ID: e6380a8e2b214edb8af41173775f711c
I20260926 01:13:51.411405 19051 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 01:13:51.441222 19051 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:13:51.441794 19051 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:13:51.450309 19051 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4935712ba089437f8cf2abbed64d693f: Generated new TSK 0
I20260926 01:13:51.450517 19051 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 01:13:51.453677 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384993574762-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384993574762-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384993574762-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384993574762-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40793
--builtin_ntp_servers=127.0.98.20:39289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 01:13:51.535877 19061 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:13:51.536093 19061 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:13:51.536114 19061 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:13:51.537302 19061 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:13:51.537348 19061 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:13:51.537374 19061 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 01:13:51.538806 19061 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384993574762-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384993574762-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384993574762-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40793
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19061
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384993574762-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:13:51.539038 19061 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:13:51.539331 19061 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:13:51.544466 19067 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:13:51.544459 19066 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:13:51.544461 19069 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:13:51.544699 19061 server_base.cc:1034] running on GCE node
I20260926 01:13:51.544912 19061 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:13:51.545145 19061 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:13:51.546300 19061 hybrid_clock.cc:648] HybridClock initialized: now 1790385231546287 us; error 27 us; skew 500 ppm
I20260926 01:13:51.547549 19061 webserver.cc:460] Webserver started at http://127.0.98.1:39895/ using document root <none> and password file <none>
I20260926 01:13:51.547808 19061 fs_manager.cc:359] Metadata directory not provided
I20260926 01:13:51.547858 19061 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:13:51.547979 19061 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:13:51.549046 19061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384993574762-392-0/minicluster-data/ts-0/data/instance:
uuid: "3ca3bad8959948a68cb01d07329ebf2f"
format_stamp: "Formatted at 2026-09-26 01:13:51 on dist-test-slave-7b22"
server_key: "4af327a76cf051c8d3774cf96c997e60"
server_key_iv: "5636ecbe69a270ce83457cb9561a457b"
server_key_version: "encryptionkey@0"
I20260926 01:13:51.549367 19061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384993574762-392-0/minicluster-data/ts-0/wal/instance:
uuid: "3ca3bad8959948a68cb01d07329ebf2f"
format_stamp: "Formatted at 2026-09-26 01:13:51 on dist-test-slave-7b22"
server_key: "4af327a76cf051c8d3774cf96c997e60"
server_key_iv: "5636ecbe69a270ce83457cb9561a457b"
server_key_version: "encryptionkey@0"
I20260926 01:13:51.550681 19061 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 01:13:51.551344 19075 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:13:51.551539 19061 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 01:13:51.551774 19061 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384993574762-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384993574762-392-0/minicluster-data/ts-0/wal
uuid: "3ca3bad8959948a68cb01d07329ebf2f"
format_stamp: "Formatted at 2026-09-26 01:13:51 on dist-test-slave-7b22"
server_key: "4af327a76cf051c8d3774cf96c997e60"
server_key_iv: "5636ecbe69a270ce83457cb9561a457b"
server_key_version: "encryptionkey@0"
I20260926 01:13:51.551851 19061 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384993574762-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384993574762-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384993574762-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:13:51.579888 19061 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:13:51.580207 19061 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:13:51.580335 19061 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:13:51.580547 19061 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:13:51.580886 19061 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:13:51.580920 19061 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:13:51.580950 19061 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:13:51.580976 19061 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:13:51.586174 19061 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44905
I20260926 01:13:51.586237 19188 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44905 every 8 connection(s)
I20260926 01:13:51.586539 19061 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384993574762-392-0/minicluster-data/ts-0/data/info.pb
I20260926 01:13:51.589177 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 19061
I20260926 01:13:51.589286 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384993574762-392-0/minicluster-data/ts-0/wal/instance
I20260926 01:13:51.589459 392 external_mini_cluster.cc:1428] Setting key 60d90d8d46da7be2f95d66d346b3544a
I20260926 01:13:51.590365 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384993574762-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384993574762-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384993574762-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384993574762-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40793
--builtin_ntp_servers=127.0.98.20:39289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 01:13:51.591859 19040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:51.587290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49433 (local address 127.0.98.62:40793)
0926 01:13:51.587346 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 01:13:51.587348 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:13:51.587559 (+ 211us) server_negotiation.cc:407] Connection header received
0926 01:13:51.588123 (+ 564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:13:51.588125 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:13:51.588142 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:13:51.588212 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:13:51.588878 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:51.589431 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:51.590451 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:51.590694 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:51.590751 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:51.591619 (+ 868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:13:51.591621 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:13:51.591621 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:13:51.591625 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:13:51.591640 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:13:51.591644 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:13:51.591683 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0926 01:13:51.591773 (+ 90us) server_negotiation.cc:299] Negotiation successful
0926 01:13:51.591799 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 01:13:51.592110 19191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:51.587252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40793 (local address 127.0.98.1:49433)
0926 01:13:51.587443 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0926 01:13:51.587477 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 01:13:51.588018 (+ 541us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:13:51.588281 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:13:51.588285 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:13:51.588293 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:13:51.588787 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:13:51.588791 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:13:51.589517 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:13:51.589518 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:13:51.590367 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:13:51.590370 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:13:51.590780 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:13:51.590782 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:13:51.590908 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:51.591441 (+ 533us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 01:13:51.591442 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:13:51.591486 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 01:13:51.591519 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:13:51.591699 (+ 180us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:13:51.591704 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:13:51.591707 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 01:13:51.591776 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 01:13:51.591888 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":88,"threads_started":1}
I20260926 01:13:51.592624 19189 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40793
I20260926 01:13:51.592713 19189 heartbeater.cc:464] Registering TS with master...
I20260926 01:13:51.592924 19189 heartbeater.cc:511] Master 127.0.98.62:40793 requested a full tablet report, sending...
I20260926 01:13:51.593391 19001 ts_manager.cc:194] Registered new tserver with Master: 3ca3bad8959948a68cb01d07329ebf2f (127.0.98.1:44905)
I20260926 01:13:51.594023 19001 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:49433
W20260926 01:13:51.671629 19192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:13:51.671859 19192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:13:51.671888 19192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:13:51.673141 19192 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:13:51.673195 19192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:13:51.673230 19192 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 01:13:51.674687 19192 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384993574762-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384993574762-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384993574762-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40793
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19192
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384993574762-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:13:51.674942 19192 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:13:51.675275 19192 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:13:51.680347 19197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:13:51.680346 19198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:13:51.680435 19192 server_base.cc:1034] running on GCE node
W20260926 01:13:51.680347 19200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:13:51.680845 19192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:13:51.681084 19192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:13:51.682256 19192 hybrid_clock.cc:648] HybridClock initialized: now 1790385231682241 us; error 32 us; skew 500 ppm
I20260926 01:13:51.690135 19192 webserver.cc:460] Webserver started at http://127.0.98.2:39203/ using document root <none> and password file <none>
I20260926 01:13:51.690428 19192 fs_manager.cc:359] Metadata directory not provided
I20260926 01:13:51.690486 19192 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:13:51.690630 19192 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:13:51.691784 19192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384993574762-392-0/minicluster-data/ts-1/data/instance:
uuid: "a35c878b3a3f4069a3fb5772018d6fae"
format_stamp: "Formatted at 2026-09-26 01:13:51 on dist-test-slave-7b22"
server_key: "347ac244a428deb4eee56cfa0dad6616"
server_key_iv: "ff24f36a6f078f3043210a7425dd487e"
server_key_version: "encryptionkey@0"
I20260926 01:13:51.692138 19192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384993574762-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a35c878b3a3f4069a3fb5772018d6fae"
format_stamp: "Formatted at 2026-09-26 01:13:51 on dist-test-slave-7b22"
server_key: "347ac244a428deb4eee56cfa0dad6616"
server_key_iv: "ff24f36a6f078f3043210a7425dd487e"
server_key_version: "encryptionkey@0"
I20260926 01:13:51.693499 19192 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 01:13:51.694531 19206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:13:51.694835 19192 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 01:13:51.694972 19192 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384993574762-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384993574762-392-0/minicluster-data/ts-1/wal
uuid: "a35c878b3a3f4069a3fb5772018d6fae"
format_stamp: "Formatted at 2026-09-26 01:13:51 on dist-test-slave-7b22"
server_key: "347ac244a428deb4eee56cfa0dad6616"
server_key_iv: "ff24f36a6f078f3043210a7425dd487e"
server_key_version: "encryptionkey@0"
I20260926 01:13:51.695106 19192 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384993574762-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384993574762-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384993574762-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:13:51.733378 19192 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:13:51.733736 19192 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:13:51.733916 19192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:13:51.734154 19192 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:13:51.734484 19192 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:13:51.734519 19192 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:13:51.734548 19192 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:13:51.734565 19192 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:13:51.739917 19192 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33061
I20260926 01:13:51.739964 19319 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33061 every 8 connection(s)
I20260926 01:13:51.740293 19192 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384993574762-392-0/minicluster-data/ts-1/data/info.pb
I20260926 01:13:51.745563 19040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:51.740993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60889 (local address 127.0.98.62:40793)
0926 01:13:51.741043 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 01:13:51.741045 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:13:51.741306 (+ 261us) server_negotiation.cc:407] Connection header received
0926 01:13:51.741880 (+ 574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:13:51.741882 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:13:51.741894 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:13:51.741975 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:13:51.742710 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:51.743262 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:51.744259 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:51.744452 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:51.744516 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:51.745335 (+ 819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:13:51.745336 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:13:51.745337 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:13:51.745342 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:13:51.745357 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:13:51.745361 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:13:51.745407 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0926 01:13:51.745475 (+ 68us) server_negotiation.cc:299] Negotiation successful
0926 01:13:51.745499 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 01:13:51.745800 19322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:51.740966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40793 (local address 127.0.98.2:60889)
0926 01:13:51.741148 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0926 01:13:51.741197 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 01:13:51.741701 (+ 504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:13:51.742075 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:13:51.742080 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:13:51.742088 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:13:51.742612 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:13:51.742617 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:13:51.743356 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:13:51.743357 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:13:51.744153 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:13:51.744155 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:13:51.744537 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:13:51.744537 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:13:51.744661 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:51.745197 (+ 536us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 01:13:51.745197 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:13:51.745228 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 01:13:51.745252 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:13:51.745415 (+ 163us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:13:51.745418 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:13:51.745421 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 01:13:51.745473 (+ 52us) client_negotiation.cc:240] Negotiation successful
0926 01:13:51.745561 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":68,"threads_started":1}
I20260926 01:13:51.746078 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 19192
I20260926 01:13:51.746191 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384993574762-392-0/minicluster-data/ts-1/wal/instance
I20260926 01:13:51.746299 19320 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40793
I20260926 01:13:51.746393 392 external_mini_cluster.cc:1428] Setting key 1e50e86e8e02f49ec4cf46d027874c3c
I20260926 01:13:51.746397 19320 heartbeater.cc:464] Registering TS with master...
I20260926 01:13:51.746587 19320 heartbeater.cc:511] Master 127.0.98.62:40793 requested a full tablet report, sending...
I20260926 01:13:51.747031 19000 ts_manager.cc:194] Registered new tserver with Master: a35c878b3a3f4069a3fb5772018d6fae (127.0.98.2:33061)
I20260926 01:13:51.747349 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384993574762-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384993574762-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384993574762-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384993574762-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40793
--builtin_ntp_servers=127.0.98.20:39289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 01:13:51.747510 19000 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:60889
W20260926 01:13:51.829093 19323 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:13:51.829334 19323 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:13:51.829360 19323 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:13:51.830713 19323 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:13:51.830773 19323 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:13:51.830806 19323 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 01:13:51.832273 19323 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384993574762-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384993574762-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384993574762-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40793
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19323
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384993574762-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:13:51.832541 19323 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:13:51.832855 19323 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:13:51.837873 19328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:13:51.837880 19329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:13:51.838188 19331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:13:51.838308 19323 server_base.cc:1034] running on GCE node
I20260926 01:13:51.838541 19323 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:13:51.838788 19323 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:13:51.839928 19323 hybrid_clock.cc:648] HybridClock initialized: now 1790385231839919 us; error 31 us; skew 500 ppm
I20260926 01:13:51.841190 19323 webserver.cc:460] Webserver started at http://127.0.98.3:34573/ using document root <none> and password file <none>
I20260926 01:13:51.841410 19323 fs_manager.cc:359] Metadata directory not provided
I20260926 01:13:51.841457 19323 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:13:51.841583 19323 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:13:51.842780 19323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384993574762-392-0/minicluster-data/ts-2/data/instance:
uuid: "98aca469458c4e2dbe5895d4f4d2e39b"
format_stamp: "Formatted at 2026-09-26 01:13:51 on dist-test-slave-7b22"
server_key: "10a11ed04babb9e45c83f011093ccbc5"
server_key_iv: "5132a4e534830831bbc06dbfe82fa12c"
server_key_version: "encryptionkey@0"
I20260926 01:13:51.843182 19323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384993574762-392-0/minicluster-data/ts-2/wal/instance:
uuid: "98aca469458c4e2dbe5895d4f4d2e39b"
format_stamp: "Formatted at 2026-09-26 01:13:51 on dist-test-slave-7b22"
server_key: "10a11ed04babb9e45c83f011093ccbc5"
server_key_iv: "5132a4e534830831bbc06dbfe82fa12c"
server_key_version: "encryptionkey@0"
I20260926 01:13:51.844429 19323 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:13:51.845155 19337 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:13:51.845309 19323 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 01:13:51.845382 19323 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384993574762-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384993574762-392-0/minicluster-data/ts-2/wal
uuid: "98aca469458c4e2dbe5895d4f4d2e39b"
format_stamp: "Formatted at 2026-09-26 01:13:51 on dist-test-slave-7b22"
server_key: "10a11ed04babb9e45c83f011093ccbc5"
server_key_iv: "5132a4e534830831bbc06dbfe82fa12c"
server_key_version: "encryptionkey@0"
I20260926 01:13:51.845451 19323 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384993574762-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384993574762-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384993574762-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:13:51.858747 19323 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:13:51.859068 19323 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:13:51.859198 19323 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:13:51.859428 19323 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:13:51.859784 19323 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:13:51.859819 19323 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:13:51.859850 19323 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:13:51.859867 19323 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:13:51.865338 19323 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37755
I20260926 01:13:51.865407 19450 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37755 every 8 connection(s)
I20260926 01:13:51.865712 19323 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384993574762-392-0/minicluster-data/ts-2/data/info.pb
I20260926 01:13:51.871207 19040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:51.866484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50875 (local address 127.0.98.62:40793)
0926 01:13:51.866556 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0926 01:13:51.866558 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:13:51.866747 (+ 189us) server_negotiation.cc:407] Connection header received
0926 01:13:51.867290 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:13:51.867292 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:13:51.867301 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:13:51.867359 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:13:51.868041 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:51.868624 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:51.869728 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:51.870032 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:51.870103 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:51.870976 (+ 873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:13:51.870977 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:13:51.870977 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:13:51.870982 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:13:51.870997 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:13:51.871002 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:13:51.871042 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0926 01:13:51.871100 (+ 58us) server_negotiation.cc:299] Negotiation successful
0926 01:13:51.871125 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 01:13:51.871379 19453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:51.866446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40793 (local address 127.0.98.3:50875)
0926 01:13:51.866628 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0926 01:13:51.866666 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 01:13:51.867168 (+ 502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:13:51.867441 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:13:51.867444 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:13:51.867452 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:13:51.867937 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:13:51.867941 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:13:51.868741 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:13:51.868742 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:13:51.869563 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:13:51.869566 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:13:51.870136 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:13:51.870137 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:13:51.870273 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:51.870836 (+ 563us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 01:13:51.870836 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:13:51.870866 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 01:13:51.870891 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:13:51.871044 (+ 153us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:13:51.871047 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:13:51.871049 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 01:13:51.871094 (+ 45us) client_negotiation.cc:240] Negotiation successful
0926 01:13:51.871183 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":75,"threads_started":1}
I20260926 01:13:51.871866 19451 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40793
I20260926 01:13:51.871959 19451 heartbeater.cc:464] Registering TS with master...
I20260926 01:13:51.872152 19451 heartbeater.cc:511] Master 127.0.98.62:40793 requested a full tablet report, sending...
I20260926 01:13:51.872305 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 19323
I20260926 01:13:51.872406 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384993574762-392-0/minicluster-data/ts-2/wal/instance
I20260926 01:13:51.872560 19000 ts_manager.cc:194] Registered new tserver with Master: 98aca469458c4e2dbe5895d4f4d2e39b (127.0.98.3:37755)
I20260926 01:13:51.872599 392 external_mini_cluster.cc:1428] Setting key 3a8b34fa618193ce76a9da3b2316e1ef
I20260926 01:13:51.873032 19000 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:50875
I20260926 01:13:51.873415 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 01:13:51.877784 19040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:51.874878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52220 (local address 127.0.98.62:40793)
0926 01:13:51.874919 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 01:13:51.874921 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:13:51.875036 (+ 115us) server_negotiation.cc:407] Connection header received
0926 01:13:51.875086 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:13:51.875087 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:13:51.875096 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:13:51.875134 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:13:51.875260 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:51.875754 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:51.876518 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:51.876716 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:51.876780 (+ 64us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:51.877501 (+ 721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:13:51.877503 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:13:51.877503 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:13:51.877508 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:13:51.877523 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:13:51.877527 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:13:51.877609 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 01:13:51.877680 (+ 71us) server_negotiation.cc:299] Negotiation successful
0926 01:13:51.877694 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 01:13:51.879513 19001 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52220:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:13:51.881088 19001 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:13:51.887540 19459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:51.883361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37755 (local address 127.0.0.1:53618)
0926 01:13:51.883865 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0926 01:13:51.883871 (+ 6us) client_negotiation.cc:174] Beginning negotiation
0926 01:13:51.883895 (+ 24us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:13:51.884295 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:13:51.884296 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:13:51.884299 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:13:51.884340 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:13:51.884341 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:13:51.885191 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:13:51.885192 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:13:51.886491 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:13:51.886493 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:13:51.887224 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:13:51.887225 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:13:51.887314 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:51.887329 (+ 15us) client_negotiation.cc:769] Sending connection context
0926 01:13:51.887394 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 01:13:51.887453 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":130,"threads_started":1}
I20260926 01:13:51.887625 19461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:51.883342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44905 (local address 127.0.0.1:38444)
0926 01:13:51.883551 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0926 01:13:51.883607 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 01:13:51.883679 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:13:51.883984 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:13:51.883987 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:13:51.883990 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:13:51.884066 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:13:51.884068 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:13:51.885444 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:13:51.885445 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:13:51.886465 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:13:51.886468 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:13:51.887394 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:13:51.887394 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:13:51.887448 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:51.887451 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 01:13:51.887488 (+ 37us) client_negotiation.cc:240] Negotiation successful
0926 01:13:51.887520 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"mutex_wait_us":40,"spinlock_wait_cycles":768,"thread_start_us":67,"threads_started":1}
I20260926 01:13:51.887645 19462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:51.883521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38444 (local address 127.0.98.1:44905)
0926 01:13:51.883817 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0926 01:13:51.883821 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 01:13:51.883830 (+ 9us) server_negotiation.cc:407] Connection header received
0926 01:13:51.883871 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:13:51.883873 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:13:51.883890 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:13:51.883969 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:13:51.884159 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:51.885355 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:51.886547 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:51.887315 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:51.887394 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:51.887486 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 01:13:51.887512 (+ 26us) server_negotiation.cc:299] Negotiation successful
0926 01:13:51.887555 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":84,"threads_started":1}
I20260926 01:13:51.887894 19464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:51.883625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53618 (local address 127.0.98.3:37755)
0926 01:13:51.884091 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0926 01:13:51.884095 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 01:13:51.884103 (+ 8us) server_negotiation.cc:407] Connection header received
0926 01:13:51.884139 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:13:51.884141 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:13:51.884154 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:13:51.884225 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:13:51.884412 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:51.885111 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:51.886592 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:51.887138 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:51.887205 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:51.887307 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 01:13:51.887770 (+ 463us) server_negotiation.cc:299] Negotiation successful
0926 01:13:51.887816 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":73,"threads_started":1}
I20260926 01:13:51.887969 19460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:51.883342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33061 (local address 127.0.0.1:49718)
0926 01:13:51.883759 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0926 01:13:51.883768 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 01:13:51.883817 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:13:51.884084 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:13:51.884085 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:13:51.884086 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:13:51.884115 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:13:51.884115 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:13:51.885432 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:13:51.885433 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:13:51.886582 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:13:51.886584 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:13:51.887520 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:13:51.887521 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:13:51.887580 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:51.887583 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 01:13:51.887853 (+ 270us) client_negotiation.cc:240] Negotiation successful
0926 01:13:51.887900 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":404,"mutex_wait_us":32,"thread_start_us":62,"threads_started":1}
I20260926 01:13:51.888010 19463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:51.883520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49718 (local address 127.0.98.2:33061)
0926 01:13:51.883732 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0926 01:13:51.883737 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 01:13:51.883820 (+ 83us) server_negotiation.cc:407] Connection header received
0926 01:13:51.883879 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:13:51.883881 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:13:51.883895 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:13:51.883969 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:13:51.884191 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:51.885342 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:51.886657 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:51.887344 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:51.887402 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:51.887485 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 01:13:51.887890 (+ 405us) server_negotiation.cc:299] Negotiation successful
0926 01:13:51.887936 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":93,"threads_started":1}
I20260926 01:13:51.888721 19254 tablet_service.cc:1467] Processing CreateTablet for tablet 994837d5df6d431dbddce83dbc378a51 (DEFAULT_TABLE table=test-table [id=a94d890a4036430fb5716b7e23c684c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:13:51.888674 19385 tablet_service.cc:1467] Processing CreateTablet for tablet 994837d5df6d431dbddce83dbc378a51 (DEFAULT_TABLE table=test-table [id=a94d890a4036430fb5716b7e23c684c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:13:51.888674 19123 tablet_service.cc:1467] Processing CreateTablet for tablet 994837d5df6d431dbddce83dbc378a51 (DEFAULT_TABLE table=test-table [id=a94d890a4036430fb5716b7e23c684c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:13:51.889066 19123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 994837d5df6d431dbddce83dbc378a51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:13:51.889066 19385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 994837d5df6d431dbddce83dbc378a51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:13:51.889079 19254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 994837d5df6d431dbddce83dbc378a51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:13:51.892011 19465 tablet_bootstrap.cc:492] T 994837d5df6d431dbddce83dbc378a51 P a35c878b3a3f4069a3fb5772018d6fae: Bootstrap starting.
I20260926 01:13:51.892450 19466 tablet_bootstrap.cc:492] T 994837d5df6d431dbddce83dbc378a51 P 3ca3bad8959948a68cb01d07329ebf2f: Bootstrap starting.
I20260926 01:13:51.892794 19465 tablet_bootstrap.cc:654] T 994837d5df6d431dbddce83dbc378a51 P a35c878b3a3f4069a3fb5772018d6fae: Neither blocks nor log segments found. Creating new log.
I20260926 01:13:51.892819 19467 tablet_bootstrap.cc:492] T 994837d5df6d431dbddce83dbc378a51 P 98aca469458c4e2dbe5895d4f4d2e39b: Bootstrap starting.
I20260926 01:13:51.893118 19466 tablet_bootstrap.cc:654] T 994837d5df6d431dbddce83dbc378a51 P 3ca3bad8959948a68cb01d07329ebf2f: Neither blocks nor log segments found. Creating new log.
I20260926 01:13:51.893111 19465 log.cc:824] T 994837d5df6d431dbddce83dbc378a51 P a35c878b3a3f4069a3fb5772018d6fae: Log is configured to *not* fsync() on all Append() calls
I20260926 01:13:51.893384 19467 tablet_bootstrap.cc:654] T 994837d5df6d431dbddce83dbc378a51 P 98aca469458c4e2dbe5895d4f4d2e39b: Neither blocks nor log segments found. Creating new log.
I20260926 01:13:51.893428 19466 log.cc:824] T 994837d5df6d431dbddce83dbc378a51 P 3ca3bad8959948a68cb01d07329ebf2f: Log is configured to *not* fsync() on all Append() calls
I20260926 01:13:51.893687 19467 log.cc:824] T 994837d5df6d431dbddce83dbc378a51 P 98aca469458c4e2dbe5895d4f4d2e39b: Log is configured to *not* fsync() on all Append() calls
I20260926 01:13:51.893975 19465 tablet_bootstrap.cc:492] T 994837d5df6d431dbddce83dbc378a51 P a35c878b3a3f4069a3fb5772018d6fae: No bootstrap required, opened a new log
I20260926 01:13:51.894054 19465 ts_tablet_manager.cc:1397] T 994837d5df6d431dbddce83dbc378a51 P a35c878b3a3f4069a3fb5772018d6fae: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 01:13:51.894214 19466 tablet_bootstrap.cc:492] T 994837d5df6d431dbddce83dbc378a51 P 3ca3bad8959948a68cb01d07329ebf2f: No bootstrap required, opened a new log
I20260926 01:13:51.894306 19466 ts_tablet_manager.cc:1397] T 994837d5df6d431dbddce83dbc378a51 P 3ca3bad8959948a68cb01d07329ebf2f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 01:13:51.894353 19467 tablet_bootstrap.cc:492] T 994837d5df6d431dbddce83dbc378a51 P 98aca469458c4e2dbe5895d4f4d2e39b: No bootstrap required, opened a new log
I20260926 01:13:51.894418 19467 ts_tablet_manager.cc:1397] T 994837d5df6d431dbddce83dbc378a51 P 98aca469458c4e2dbe5895d4f4d2e39b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 01:13:51.896428 19467 raft_consensus.cc:348] T 994837d5df6d431dbddce83dbc378a51 P 98aca469458c4e2dbe5895d4f4d2e39b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a35c878b3a3f4069a3fb5772018d6fae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33061 } } peers { permanent_uuid: "3ca3bad8959948a68cb01d07329ebf2f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44905 } } peers { permanent_uuid: "98aca469458c4e2dbe5895d4f4d2e39b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37755 } }
I20260926 01:13:51.896580 19467 raft_consensus.cc:374] T 994837d5df6d431dbddce83dbc378a51 P 98aca469458c4e2dbe5895d4f4d2e39b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:13:51.896612 19467 raft_consensus.cc:729] T 994837d5df6d431dbddce83dbc378a51 P 98aca469458c4e2dbe5895d4f4d2e39b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98aca469458c4e2dbe5895d4f4d2e39b, State: Initialized, Role: FOLLOWER
I20260926 01:13:51.896771 19467 consensus_queue.cc:260] T 994837d5df6d431dbddce83dbc378a51 P 98aca469458c4e2dbe5895d4f4d2e39b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a35c878b3a3f4069a3fb5772018d6fae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33061 } } peers { permanent_uuid: "3ca3bad8959948a68cb01d07329ebf2f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44905 } } peers { permanent_uuid: "98aca469458c4e2dbe5895d4f4d2e39b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37755 } }
I20260926 01:13:51.896847 19465 raft_consensus.cc:348] T 994837d5df6d431dbddce83dbc378a51 P a35c878b3a3f4069a3fb5772018d6fae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a35c878b3a3f4069a3fb5772018d6fae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33061 } } peers { permanent_uuid: "3ca3bad8959948a68cb01d07329ebf2f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44905 } } peers { permanent_uuid: "98aca469458c4e2dbe5895d4f4d2e39b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37755 } }
I20260926 01:13:51.896840 19466 raft_consensus.cc:348] T 994837d5df6d431dbddce83dbc378a51 P 3ca3bad8959948a68cb01d07329ebf2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a35c878b3a3f4069a3fb5772018d6fae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33061 } } peers { permanent_uuid: "3ca3bad8959948a68cb01d07329ebf2f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44905 } } peers { permanent_uuid: "98aca469458c4e2dbe5895d4f4d2e39b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37755 } }
I20260926 01:13:51.897003 19465 raft_consensus.cc:374] T 994837d5df6d431dbddce83dbc378a51 P a35c878b3a3f4069a3fb5772018d6fae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:13:51.897003 19466 raft_consensus.cc:374] T 994837d5df6d431dbddce83dbc378a51 P 3ca3bad8959948a68cb01d07329ebf2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:13:51.897034 19467 ts_tablet_manager.cc:1428] T 994837d5df6d431dbddce83dbc378a51 P 98aca469458c4e2dbe5895d4f4d2e39b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 01:13:51.897043 19466 raft_consensus.cc:729] T 994837d5df6d431dbddce83dbc378a51 P 3ca3bad8959948a68cb01d07329ebf2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ca3bad8959948a68cb01d07329ebf2f, State: Initialized, Role: FOLLOWER
I20260926 01:13:51.897090 19465 raft_consensus.cc:729] T 994837d5df6d431dbddce83dbc378a51 P a35c878b3a3f4069a3fb5772018d6fae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a35c878b3a3f4069a3fb5772018d6fae, State: Initialized, Role: FOLLOWER
I20260926 01:13:51.897151 19451 heartbeater.cc:503] Master 127.0.98.62:40793 was elected leader, sending a full tablet report...
I20260926 01:13:51.897157 19466 consensus_queue.cc:260] T 994837d5df6d431dbddce83dbc378a51 P 3ca3bad8959948a68cb01d07329ebf2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a35c878b3a3f4069a3fb5772018d6fae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33061 } } peers { permanent_uuid: "3ca3bad8959948a68cb01d07329ebf2f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44905 } } peers { permanent_uuid: "98aca469458c4e2dbe5895d4f4d2e39b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37755 } }
I20260926 01:13:51.897259 19465 consensus_queue.cc:260] T 994837d5df6d431dbddce83dbc378a51 P a35c878b3a3f4069a3fb5772018d6fae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a35c878b3a3f4069a3fb5772018d6fae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33061 } } peers { permanent_uuid: "3ca3bad8959948a68cb01d07329ebf2f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44905 } } peers { permanent_uuid: "98aca469458c4e2dbe5895d4f4d2e39b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37755 } }
I20260926 01:13:51.897419 19466 ts_tablet_manager.cc:1428] T 994837d5df6d431dbddce83dbc378a51 P 3ca3bad8959948a68cb01d07329ebf2f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 01:13:51.897468 19465 ts_tablet_manager.cc:1428] T 994837d5df6d431dbddce83dbc378a51 P a35c878b3a3f4069a3fb5772018d6fae: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 01:13:51.897570 19320 heartbeater.cc:503] Master 127.0.98.62:40793 was elected leader, sending a full tablet report...
I20260926 01:13:51.897933 19189 heartbeater.cc:503] Master 127.0.98.62:40793 was elected leader, sending a full tablet report...
W20260926 01:13:51.990869 19321 tablet.cc:2369] T 994837d5df6d431dbddce83dbc378a51 P a35c878b3a3f4069a3fb5772018d6fae: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 01:13:52.087262 19190 tablet.cc:2369] T 994837d5df6d431dbddce83dbc378a51 P 3ca3bad8959948a68cb01d07329ebf2f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 01:13:52.116294 19452 tablet.cc:2369] T 994837d5df6d431dbddce83dbc378a51 P 98aca469458c4e2dbe5895d4f4d2e39b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 01:13:52.206152 19472 raft_consensus.cc:482] T 994837d5df6d431dbddce83dbc378a51 P 3ca3bad8959948a68cb01d07329ebf2f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:13:52.206305 19472 raft_consensus.cc:504] T 994837d5df6d431dbddce83dbc378a51 P 3ca3bad8959948a68cb01d07329ebf2f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a35c878b3a3f4069a3fb5772018d6fae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33061 } } peers { permanent_uuid: "3ca3bad8959948a68cb01d07329ebf2f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44905 } } peers { permanent_uuid: "98aca469458c4e2dbe5895d4f4d2e39b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37755 } }
I20260926 01:13:52.206784 19472 leader_election.cc:288] T 994837d5df6d431dbddce83dbc378a51 P 3ca3bad8959948a68cb01d07329ebf2f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a35c878b3a3f4069a3fb5772018d6fae (127.0.98.2:33061), 98aca469458c4e2dbe5895d4f4d2e39b (127.0.98.3:37755)
I20260926 01:13:52.209913 19463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:52.207128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44781 (local address 127.0.98.2:33061)
0926 01:13:52.207177 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 01:13:52.207179 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:13:52.207271 (+ 92us) server_negotiation.cc:407] Connection header received
0926 01:13:52.207309 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:13:52.207310 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:13:52.207322 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:13:52.207375 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:13:52.207532 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:52.208051 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:52.209152 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:52.209555 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:52.209643 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:52.209654 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 01:13:52.209770 (+ 116us) server_negotiation.cc:299] Negotiation successful
0926 01:13:52.209805 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 01:13:52.209930 19475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:52.207018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33061 (local address 127.0.98.1:44781)
0926 01:13:52.207194 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 01:13:52.207205 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 01:13:52.207264 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:13:52.207385 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:13:52.207386 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:13:52.207388 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:13:52.207471 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:13:52.207474 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:13:52.208146 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:13:52.208146 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:13:52.209063 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:13:52.209066 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:13:52.209625 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:13:52.209626 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:13:52.209696 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:52.209700 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 01:13:52.209763 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 01:13:52.209840 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"mutex_wait_us":37,"thread_start_us":67,"threads_started":1}
I20260926 01:13:52.210155 19464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:52.207113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39493 (local address 127.0.98.3:37755)
0926 01:13:52.207170 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 01:13:52.207172 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:13:52.207305 (+ 133us) server_negotiation.cc:407] Connection header received
0926 01:13:52.207370 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:13:52.207371 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:13:52.207381 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:13:52.207432 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:13:52.207577 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:52.208041 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:52.209156 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:52.209748 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:52.209802 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:52.209811 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 01:13:52.210031 (+ 220us) server_negotiation.cc:299] Negotiation successful
0926 01:13:52.210079 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 01:13:52.210260 19274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "994837d5df6d431dbddce83dbc378a51" candidate_uuid: "3ca3bad8959948a68cb01d07329ebf2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a35c878b3a3f4069a3fb5772018d6fae" is_pre_election: true
I20260926 01:13:52.210386 19474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:52.207022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37755 (local address 127.0.98.1:39493)
0926 01:13:52.207209 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0926 01:13:52.207219 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 01:13:52.207302 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:13:52.207441 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:13:52.207442 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:13:52.207444 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:13:52.207516 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:13:52.207518 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:13:52.208120 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:13:52.208122 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:13:52.209053 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:13:52.209055 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:13:52.209872 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:13:52.209873 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:13:52.209939 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:52.209943 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 01:13:52.210263 (+ 320us) client_negotiation.cc:240] Negotiation successful
0926 01:13:52.210324 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":68,"threads_started":1}
I20260926 01:13:52.210448 19274 raft_consensus.cc:2454] T 994837d5df6d431dbddce83dbc378a51 P a35c878b3a3f4069a3fb5772018d6fae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ca3bad8959948a68cb01d07329ebf2f in term 0.
I20260926 01:13:52.210669 19077 leader_election.cc:302] T 994837d5df6d431dbddce83dbc378a51 P 3ca3bad8959948a68cb01d07329ebf2f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ca3bad8959948a68cb01d07329ebf2f, a35c878b3a3f4069a3fb5772018d6fae; no voters:
I20260926 01:13:52.210727 19405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "994837d5df6d431dbddce83dbc378a51" candidate_uuid: "3ca3bad8959948a68cb01d07329ebf2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98aca469458c4e2dbe5895d4f4d2e39b" is_pre_election: true
I20260926 01:13:52.210809 19472 raft_consensus.cc:2781] T 994837d5df6d431dbddce83dbc378a51 P 3ca3bad8959948a68cb01d07329ebf2f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:13:52.210871 19472 raft_consensus.cc:482] T 994837d5df6d431dbddce83dbc378a51 P 3ca3bad8959948a68cb01d07329ebf2f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:13:52.210870 19405 raft_consensus.cc:2454] T 994837d5df6d431dbddce83dbc378a51 P 98aca469458c4e2dbe5895d4f4d2e39b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ca3bad8959948a68cb01d07329ebf2f in term 0.
I20260926 01:13:52.210894 19472 raft_consensus.cc:3037] T 994837d5df6d431dbddce83dbc378a51 P 3ca3bad8959948a68cb01d07329ebf2f [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:13:52.211630 19472 raft_consensus.cc:504] T 994837d5df6d431dbddce83dbc378a51 P 3ca3bad8959948a68cb01d07329ebf2f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a35c878b3a3f4069a3fb5772018d6fae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33061 } } peers { permanent_uuid: "3ca3bad8959948a68cb01d07329ebf2f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44905 } } peers { permanent_uuid: "98aca469458c4e2dbe5895d4f4d2e39b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37755 } }
I20260926 01:13:52.211759 19472 leader_election.cc:288] T 994837d5df6d431dbddce83dbc378a51 P 3ca3bad8959948a68cb01d07329ebf2f [CANDIDATE]: Term 1 election: Requested vote from peers a35c878b3a3f4069a3fb5772018d6fae (127.0.98.2:33061), 98aca469458c4e2dbe5895d4f4d2e39b (127.0.98.3:37755)
I20260926 01:13:52.211898 19405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "994837d5df6d431dbddce83dbc378a51" candidate_uuid: "3ca3bad8959948a68cb01d07329ebf2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98aca469458c4e2dbe5895d4f4d2e39b"
I20260926 01:13:52.211963 19274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "994837d5df6d431dbddce83dbc378a51" candidate_uuid: "3ca3bad8959948a68cb01d07329ebf2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a35c878b3a3f4069a3fb5772018d6fae"
I20260926 01:13:52.211975 19405 raft_consensus.cc:3037] T 994837d5df6d431dbddce83dbc378a51 P 98aca469458c4e2dbe5895d4f4d2e39b [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:13:52.212051 19274 raft_consensus.cc:3037] T 994837d5df6d431dbddce83dbc378a51 P a35c878b3a3f4069a3fb5772018d6fae [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:13:52.212661 19405 raft_consensus.cc:2454] T 994837d5df6d431dbddce83dbc378a51 P 98aca469458c4e2dbe5895d4f4d2e39b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ca3bad8959948a68cb01d07329ebf2f in term 1.
I20260926 01:13:52.212828 19274 raft_consensus.cc:2454] T 994837d5df6d431dbddce83dbc378a51 P a35c878b3a3f4069a3fb5772018d6fae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ca3bad8959948a68cb01d07329ebf2f in term 1.
I20260926 01:13:52.212833 19078 leader_election.cc:302] T 994837d5df6d431dbddce83dbc378a51 P 3ca3bad8959948a68cb01d07329ebf2f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ca3bad8959948a68cb01d07329ebf2f, 98aca469458c4e2dbe5895d4f4d2e39b; no voters:
I20260926 01:13:52.212922 19472 raft_consensus.cc:2781] T 994837d5df6d431dbddce83dbc378a51 P 3ca3bad8959948a68cb01d07329ebf2f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:13:52.213074 19472 raft_consensus.cc:686] T 994837d5df6d431dbddce83dbc378a51 P 3ca3bad8959948a68cb01d07329ebf2f [term 1 LEADER]: Becoming Leader. State: Replica: 3ca3bad8959948a68cb01d07329ebf2f, State: Running, Role: LEADER
I20260926 01:13:52.213171 19472 consensus_queue.cc:237] T 994837d5df6d431dbddce83dbc378a51 P 3ca3bad8959948a68cb01d07329ebf2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a35c878b3a3f4069a3fb5772018d6fae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33061 } } peers { permanent_uuid: "3ca3bad8959948a68cb01d07329ebf2f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44905 } } peers { permanent_uuid: "98aca469458c4e2dbe5895d4f4d2e39b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37755 } }
I20260926 01:13:52.213994 19000 catalog_manager.cc:5495] T 994837d5df6d431dbddce83dbc378a51 P 3ca3bad8959948a68cb01d07329ebf2f reported cstate change: term changed from 0 to 1, leader changed from <none> to 3ca3bad8959948a68cb01d07329ebf2f (127.0.98.1). New cstate: current_term: 1 leader_uuid: "3ca3bad8959948a68cb01d07329ebf2f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a35c878b3a3f4069a3fb5772018d6fae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33061 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3ca3bad8959948a68cb01d07329ebf2f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44905 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98aca469458c4e2dbe5895d4f4d2e39b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37755 } health_report { overall_health: UNKNOWN } } }
I20260926 01:13:52.241091 19462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:52.238814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38460 (local address 127.0.98.1:44905)
0926 01:13:52.238867 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 01:13:52.238869 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:13:52.238877 (+ 8us) server_negotiation.cc:407] Connection header received
0926 01:13:52.238922 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:13:52.238923 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:13:52.238944 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:13:52.239007 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:13:52.239139 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:52.239625 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:52.240347 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:52.240532 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:52.240586 (+ 54us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:52.240779 (+ 193us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:13:52.240871 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:13:52.240957 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 01:13:52.241004 (+ 47us) server_negotiation.cc:299] Negotiation successful
0926 01:13:52.241014 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 01:13:52.242959 19405 raft_consensus.cc:1261] T 994837d5df6d431dbddce83dbc378a51 P 98aca469458c4e2dbe5895d4f4d2e39b [term 1 FOLLOWER]: Refusing update from remote peer 3ca3bad8959948a68cb01d07329ebf2f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:13:52.242970 19274 raft_consensus.cc:1261] T 994837d5df6d431dbddce83dbc378a51 P a35c878b3a3f4069a3fb5772018d6fae [term 1 FOLLOWER]: Refusing update from remote peer 3ca3bad8959948a68cb01d07329ebf2f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:13:52.243333 19476 consensus_queue.cc:1035] T 994837d5df6d431dbddce83dbc378a51 P 3ca3bad8959948a68cb01d07329ebf2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "98aca469458c4e2dbe5895d4f4d2e39b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:13:52.243458 19472 consensus_queue.cc:1035] T 994837d5df6d431dbddce83dbc378a51 P 3ca3bad8959948a68cb01d07329ebf2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a35c878b3a3f4069a3fb5772018d6fae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:13:52.245842 19479 mvcc.cc:205] Tried to move back new op lower bound from 7333417911264395264 to 7333417911145623552. Current Snapshot: MvccSnapshot[applied={T|T < 7333417911264395264}]
I20260926 01:13:52.246616 19481 mvcc.cc:205] Tried to move back new op lower bound from 7333417911264395264 to 7333417911145623552. Current Snapshot: MvccSnapshot[applied={T|T < 7333417911264395264}]
I20260926 01:13:52.249054 19001 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52220:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:13:52.249163 19001 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52220:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:13:52.249891 19001 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4935712ba089437f8cf2abbed64d693f: Sending DeleteTablet for 3 replicas of tablet 994837d5df6d431dbddce83dbc378a51
I20260926 01:13:52.250272 19123 tablet_service.cc:1514] Processing DeleteTablet for tablet 994837d5df6d431dbddce83dbc378a51 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:13:52 UTC) from {username='slave'} at 127.0.0.1:38444
I20260926 01:13:52.250257 19254 tablet_service.cc:1514] Processing DeleteTablet for tablet 994837d5df6d431dbddce83dbc378a51 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:13:52 UTC) from {username='slave'} at 127.0.0.1:49718
I20260926 01:13:52.250355 19385 tablet_service.cc:1514] Processing DeleteTablet for tablet 994837d5df6d431dbddce83dbc378a51 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:13:52 UTC) from {username='slave'} at 127.0.0.1:53618
I20260926 01:13:52.250521 19489 tablet_replica.cc:330] T 994837d5df6d431dbddce83dbc378a51 P 3ca3bad8959948a68cb01d07329ebf2f: stopping tablet replica
I20260926 01:13:52.250619 19489 raft_consensus.cc:2229] T 994837d5df6d431dbddce83dbc378a51 P 3ca3bad8959948a68cb01d07329ebf2f [term 1 LEADER]: Raft consensus shutting down.
I20260926 01:13:52.250638 19488 tablet_replica.cc:330] T 994837d5df6d431dbddce83dbc378a51 P a35c878b3a3f4069a3fb5772018d6fae: stopping tablet replica
I20260926 01:13:52.250847 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 19061
I20260926 01:13:52.251413 19490 tablet_replica.cc:330] T 994837d5df6d431dbddce83dbc378a51 P 98aca469458c4e2dbe5895d4f4d2e39b: stopping tablet replica
I20260926 01:13:52.251511 19490 raft_consensus.cc:2229] T 994837d5df6d431dbddce83dbc378a51 P 98aca469458c4e2dbe5895d4f4d2e39b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:13:52.251606 19490 raft_consensus.cc:2258] T 994837d5df6d431dbddce83dbc378a51 P 98aca469458c4e2dbe5895d4f4d2e39b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:13:52.252040 19490 ts_tablet_manager.cc:1905] T 994837d5df6d431dbddce83dbc378a51 P 98aca469458c4e2dbe5895d4f4d2e39b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:13:52.253139 19488 raft_consensus.cc:2229] T 994837d5df6d431dbddce83dbc378a51 P a35c878b3a3f4069a3fb5772018d6fae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:13:52.253297 19488 raft_consensus.cc:2258] T 994837d5df6d431dbddce83dbc378a51 P a35c878b3a3f4069a3fb5772018d6fae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:13:52.253441 19490 ts_tablet_manager.cc:1918] T 994837d5df6d431dbddce83dbc378a51 P 98aca469458c4e2dbe5895d4f4d2e39b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:13:52.253518 19490 log.cc:1196] T 994837d5df6d431dbddce83dbc378a51 P 98aca469458c4e2dbe5895d4f4d2e39b: Deleting WAL directory at /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384993574762-392-0/minicluster-data/ts-2/wal/wals/994837d5df6d431dbddce83dbc378a51
I20260926 01:13:52.253782 19490 ts_tablet_manager.cc:1939] T 994837d5df6d431dbddce83dbc378a51 P 98aca469458c4e2dbe5895d4f4d2e39b: Deleting consensus metadata
I20260926 01:13:52.253799 19488 ts_tablet_manager.cc:1905] T 994837d5df6d431dbddce83dbc378a51 P a35c878b3a3f4069a3fb5772018d6fae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:13:52.254221 18987 catalog_manager.cc:4841] TS 98aca469458c4e2dbe5895d4f4d2e39b (127.0.98.3:37755): tablet 994837d5df6d431dbddce83dbc378a51 (table test-table [id=a94d890a4036430fb5716b7e23c684c8]) successfully deleted
I20260926 01:13:52.255472 19488 ts_tablet_manager.cc:1918] T 994837d5df6d431dbddce83dbc378a51 P a35c878b3a3f4069a3fb5772018d6fae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:13:52.255548 19488 log.cc:1196] T 994837d5df6d431dbddce83dbc378a51 P a35c878b3a3f4069a3fb5772018d6fae: Deleting WAL directory at /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384993574762-392-0/minicluster-data/ts-1/wal/wals/994837d5df6d431dbddce83dbc378a51
I20260926 01:13:52.255775 19488 ts_tablet_manager.cc:1939] T 994837d5df6d431dbddce83dbc378a51 P a35c878b3a3f4069a3fb5772018d6fae: Deleting consensus metadata
I20260926 01:13:52.256253 18986 catalog_manager.cc:4841] TS a35c878b3a3f4069a3fb5772018d6fae (127.0.98.2:33061): tablet 994837d5df6d431dbddce83dbc378a51 (table test-table [id=a94d890a4036430fb5716b7e23c684c8]) successfully deleted
W20260926 01:13:52.256292 18985 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44905 (error 108)
I20260926 01:13:52.256790 19459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:52.256647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44905 (local address 127.0.0.1:38470)
0926 01:13:52.256715 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0926 01:13:52.256761 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260926 01:13:52.256803 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 19192
W20260926 01:13:52.256912 18985 catalog_manager.cc:4584] TS 3ca3bad8959948a68cb01d07329ebf2f (127.0.98.1:44905): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 994837d5df6d431dbddce83dbc378a51: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44905: connect: Connection refused (error 111)
I20260926 01:13:52.261436 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 19323
I20260926 01:13:52.265569 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 18970
2026-09-26T01:13:52Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1015 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384993574762-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 01:13:52 dist-test-slave-7b22 krb5kdc[19494](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 01:13:52 dist-test-slave-7b22 krb5kdc[19494](info): set up 2 sockets
Sep 26 01:13:52 dist-test-slave-7b22 krb5kdc[19494](info): commencing operation
krb5kdc: starting...
W20260926 01:13:53.229740 392 mini_kdc.cc:121] Time spent starting KDC: real 0.946s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384993574762-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384993574762-392-0/krb5kdc/test-user.keytab.
Sep 26 01:13:53 dist-test-slave-7b22 krb5kdc[19494](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385233, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T01:13:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:13:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384993574762-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384993574762-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 01:13:53.349082 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384993574762-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384993574762-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384993574762-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39609
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384993574762-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39609
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:13:53.428638 19507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:13:53.428864 19507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:13:53.428885 19507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:13:53.430100 19507 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:13:53.430143 19507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:13:53.430156 19507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:13:53.430171 19507 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:13:53.430181 19507 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:13:53.431759 19507 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384993574762-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384993574762-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39609
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384993574762-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39609
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19507
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384993574762-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:13:53.432009 19507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:13:53.432238 19507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 01:13:53.437577 19507 server_base.cc:1034] running on GCE node
W20260926 01:13:53.437546 19513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:13:53.437537 19512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:13:53.437773 19515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:13:53.438068 19507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:13:53.438410 19507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:13:53.439568 19507 hybrid_clock.cc:648] HybridClock initialized: now 1790385233439550 us; error 39 us; skew 500 ppm
Sep 26 01:13:53 dist-test-slave-7b22 krb5kdc[19494](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385233, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:13:53.442181 19507 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 01:13:53.442677 19507 webserver.cc:460] Webserver started at http://127.0.98.62:43397/ using document root <none> and password file <none>
I20260926 01:13:53.442903 19507 fs_manager.cc:359] Metadata directory not provided
I20260926 01:13:53.442951 19507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:13:53.443085 19507 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:13:53.444170 19507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384993574762-392-0/minicluster-data/master-0/data/instance:
uuid: "c43b423c7517487bac9b8fbd3afad894"
format_stamp: "Formatted at 2026-09-26 01:13:53 on dist-test-slave-7b22"
server_key: "3e3e1bf58a3d1b033231c69156288ef2"
server_key_iv: "ea2ef82c4506e08248d1c8b2057b654a"
server_key_version: "encryptionkey@0"
I20260926 01:13:53.444509 19507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384993574762-392-0/minicluster-data/master-0/wal/instance:
uuid: "c43b423c7517487bac9b8fbd3afad894"
format_stamp: "Formatted at 2026-09-26 01:13:53 on dist-test-slave-7b22"
server_key: "3e3e1bf58a3d1b033231c69156288ef2"
server_key_iv: "ea2ef82c4506e08248d1c8b2057b654a"
server_key_version: "encryptionkey@0"
I20260926 01:13:53.445926 19507 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 01:13:53.446777 19522 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:13:53.446945 19507 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:13:53.447028 19507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384993574762-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384993574762-392-0/minicluster-data/master-0/wal
uuid: "c43b423c7517487bac9b8fbd3afad894"
format_stamp: "Formatted at 2026-09-26 01:13:53 on dist-test-slave-7b22"
server_key: "3e3e1bf58a3d1b033231c69156288ef2"
server_key_iv: "ea2ef82c4506e08248d1c8b2057b654a"
server_key_version: "encryptionkey@0"
I20260926 01:13:53.447114 19507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384993574762-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384993574762-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384993574762-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:13:53.460729 19507 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:13:53.462859 19507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:13:53.463006 19507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:13:53.467007 19507 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39609
I20260926 01:13:53.467047 19574 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39609 every 8 connection(s)
I20260926 01:13:53.467376 19507 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
I20260926 01:13:53.468282 19575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:13:53.470642 19575 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c43b423c7517487bac9b8fbd3afad894: Bootstrap starting.
I20260926 01:13:53.471215 19575 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c43b423c7517487bac9b8fbd3afad894: Neither blocks nor log segments found. Creating new log.
I20260926 01:13:53.471472 19575 log.cc:824] T 00000000000000000000000000000000 P c43b423c7517487bac9b8fbd3afad894: Log is configured to *not* fsync() on all Append() calls
I20260926 01:13:53.472137 19575 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c43b423c7517487bac9b8fbd3afad894: No bootstrap required, opened a new log
I20260926 01:13:53.473793 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 19507
I20260926 01:13:53.473923 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384993574762-392-0/minicluster-data/master-0/wal/instance
I20260926 01:13:53.473778 19575 raft_consensus.cc:348] T 00000000000000000000000000000000 P c43b423c7517487bac9b8fbd3afad894 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c43b423c7517487bac9b8fbd3afad894" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39609 } }
I20260926 01:13:53.474037 19575 raft_consensus.cc:374] T 00000000000000000000000000000000 P c43b423c7517487bac9b8fbd3afad894 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:13:53.474071 19575 raft_consensus.cc:729] T 00000000000000000000000000000000 P c43b423c7517487bac9b8fbd3afad894 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c43b423c7517487bac9b8fbd3afad894, State: Initialized, Role: FOLLOWER
I20260926 01:13:53.474118 392 external_mini_cluster.cc:1428] Setting key 141431dfa0173129181becbb7c02a4d8
I20260926 01:13:53.474252 19575 consensus_queue.cc:260] T 00000000000000000000000000000000 P c43b423c7517487bac9b8fbd3afad894 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c43b423c7517487bac9b8fbd3afad894" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39609 } }
I20260926 01:13:53.474334 19575 raft_consensus.cc:388] T 00000000000000000000000000000000 P c43b423c7517487bac9b8fbd3afad894 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:13:53.474357 19575 raft_consensus.cc:482] T 00000000000000000000000000000000 P c43b423c7517487bac9b8fbd3afad894 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:13:53.474406 19575 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c43b423c7517487bac9b8fbd3afad894 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:13:53.475339 19575 raft_consensus.cc:504] T 00000000000000000000000000000000 P c43b423c7517487bac9b8fbd3afad894 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c43b423c7517487bac9b8fbd3afad894" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39609 } }
I20260926 01:13:53.475456 19575 leader_election.cc:302] T 00000000000000000000000000000000 P c43b423c7517487bac9b8fbd3afad894 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c43b423c7517487bac9b8fbd3afad894; no voters:
I20260926 01:13:53.475616 19575 leader_election.cc:288] T 00000000000000000000000000000000 P c43b423c7517487bac9b8fbd3afad894 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:13:53.475638 19580 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c43b423c7517487bac9b8fbd3afad894 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:13:53.475790 19580 raft_consensus.cc:686] T 00000000000000000000000000000000 P c43b423c7517487bac9b8fbd3afad894 [term 1 LEADER]: Becoming Leader. State: Replica: c43b423c7517487bac9b8fbd3afad894, State: Running, Role: LEADER
I20260926 01:13:53.475883 19575 sys_catalog.cc:564] T 00000000000000000000000000000000 P c43b423c7517487bac9b8fbd3afad894 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:13:53.475920 19580 consensus_queue.cc:237] T 00000000000000000000000000000000 P c43b423c7517487bac9b8fbd3afad894 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c43b423c7517487bac9b8fbd3afad894" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39609 } }
I20260926 01:13:53.476260 19580 sys_catalog.cc:455] T 00000000000000000000000000000000 P c43b423c7517487bac9b8fbd3afad894 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c43b423c7517487bac9b8fbd3afad894. Latest consensus state: current_term: 1 leader_uuid: "c43b423c7517487bac9b8fbd3afad894" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c43b423c7517487bac9b8fbd3afad894" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39609 } } }
I20260926 01:13:53.476270 19582 sys_catalog.cc:455] T 00000000000000000000000000000000 P c43b423c7517487bac9b8fbd3afad894 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c43b423c7517487bac9b8fbd3afad894" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c43b423c7517487bac9b8fbd3afad894" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39609 } } }
I20260926 01:13:53.476420 19582 sys_catalog.cc:458] T 00000000000000000000000000000000 P c43b423c7517487bac9b8fbd3afad894 [sys.catalog]: This master's current role is: LEADER
I20260926 01:13:53.476599 19580 sys_catalog.cc:458] T 00000000000000000000000000000000 P c43b423c7517487bac9b8fbd3afad894 [sys.catalog]: This master's current role is: LEADER
W20260926 01:13:53.477517 19596 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c43b423c7517487bac9b8fbd3afad894: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 01:13:53.477617 19596 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 01:13:53.477677 19588 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:13:53.477813 19588 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:13:53.479332 19588 catalog_manager.cc:1347] Generated new cluster ID: 70593b9668d447f6b5f7312392487bbc
I20260926 01:13:53.479383 19588 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 01:13:53 dist-test-slave-7b22 krb5kdc[19494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385233, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:13:53.484452 19579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:53.474789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33726 (local address 127.0.98.62:39609)
0926 01:13:53.474982 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 01:13:53.474992 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 01:13:53.475016 (+ 24us) server_negotiation.cc:407] Connection header received
0926 01:13:53.475108 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:13:53.475113 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:13:53.475158 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:13:53.475266 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:13:53.476049 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:53.477490 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:53.478336 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:53.478783 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:53.478881 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:53.481070 (+ 2189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:13:53.481074 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:13:53.481075 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:13:53.481086 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:13:53.482611 (+ 1525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:13:53.482941 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:13:53.482945 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:13:53.482945 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:13:53.483410 (+ 465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:13:53.483596 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:13:53.483597 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:13:53.483597 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:13:53.483875 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:13:53.484020 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0926 01:13:53.484116 (+ 96us) server_negotiation.cc:299] Negotiation successful
0926 01:13:53.484254 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":107,"threads_started":1}
I20260926 01:13:53.491187 19588 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:13:53.491935 19588 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:13:53.495062 19588 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c43b423c7517487bac9b8fbd3afad894: Generated new TSK 0
I20260926 01:13:53.495254 19588 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384993574762-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384993574762-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 01:13:53.526131 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384993574762-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384993574762-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384993574762-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384993574762-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39609
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384993574762-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:13:53.608623 19601 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:13:53.608850 19601 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:13:53.608873 19601 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:13:53.610044 19601 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:13:53.610081 19601 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:13:53.610106 19601 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 01:13:53.611657 19601 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384993574762-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384993574762-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384993574762-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384993574762-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39609
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19601
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384993574762-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:13:53.611866 19601 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:13:53.612043 19601 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:13:53.617062 19607 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:13:53.617076 19601 server_base.cc:1034] running on GCE node
W20260926 01:13:53.617046 19606 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:13:53.617046 19609 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:13:53.617489 19601 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:13:53.617718 19601 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:13:53.618866 19601 hybrid_clock.cc:648] HybridClock initialized: now 1790385233618854 us; error 27 us; skew 500 ppm
Sep 26 01:13:53 dist-test-slave-7b22 krb5kdc[19494](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385233, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:13:53.621244 19601 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 01:13:53.621667 19601 webserver.cc:460] Webserver started at http://127.0.98.1:35873/ using document root <none> and password file <none>
I20260926 01:13:53.621906 19601 fs_manager.cc:359] Metadata directory not provided
I20260926 01:13:53.621956 19601 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:13:53.622079 19601 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:13:53.623160 19601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384993574762-392-0/minicluster-data/ts-0/data/instance:
uuid: "7a02b46540a548249f2cd72d6f0b51c5"
format_stamp: "Formatted at 2026-09-26 01:13:53 on dist-test-slave-7b22"
server_key: "39a9e2dbc086d0978ebe9c8254be6e66"
server_key_iv: "4c9ccfca3b15e6c525473856e3959b61"
server_key_version: "encryptionkey@0"
I20260926 01:13:53.623483 19601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384993574762-392-0/minicluster-data/ts-0/wal/instance:
uuid: "7a02b46540a548249f2cd72d6f0b51c5"
format_stamp: "Formatted at 2026-09-26 01:13:53 on dist-test-slave-7b22"
server_key: "39a9e2dbc086d0978ebe9c8254be6e66"
server_key_iv: "4c9ccfca3b15e6c525473856e3959b61"
server_key_version: "encryptionkey@0"
I20260926 01:13:53.624779 19601 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 01:13:53.625519 19616 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:13:53.625737 19601 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 01:13:53.625847 19601 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384993574762-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384993574762-392-0/minicluster-data/ts-0/wal
uuid: "7a02b46540a548249f2cd72d6f0b51c5"
format_stamp: "Formatted at 2026-09-26 01:13:53 on dist-test-slave-7b22"
server_key: "39a9e2dbc086d0978ebe9c8254be6e66"
server_key_iv: "4c9ccfca3b15e6c525473856e3959b61"
server_key_version: "encryptionkey@0"
I20260926 01:13:53.625924 19601 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384993574762-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384993574762-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384993574762-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:13:53.645942 19601 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:13:53.648056 19601 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:13:53.648193 19601 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:13:53.648413 19601 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:13:53.648718 19601 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:13:53.648773 19601 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:13:53.648801 19601 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:13:53.648813 19601 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:13:53.654470 19601 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44207
I20260926 01:13:53.654543 19729 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44207 every 8 connection(s)
I20260926 01:13:53.654834 19601 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384993574762-392-0/minicluster-data/ts-0/data/info.pb
Sep 26 01:13:53 dist-test-slave-7b22 krb5kdc[19494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385233, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:13:53.661207 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 19601
I20260926 01:13:53.661306 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384993574762-392-0/minicluster-data/ts-0/wal/instance
I20260926 01:13:53.661480 392 external_mini_cluster.cc:1428] Setting key 1383c8f1eaacfabda494b6a87e94444c
I20260926 01:13:53.664295 19579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:53.655537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60875 (local address 127.0.98.62:39609)
0926 01:13:53.655593 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 01:13:53.655595 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:13:53.655808 (+ 213us) server_negotiation.cc:407] Connection header received
0926 01:13:53.656752 (+ 944us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:13:53.656753 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:13:53.656771 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:13:53.656824 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:13:53.657440 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:53.658009 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:53.658993 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:53.659211 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:53.659268 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:53.661479 (+ 2211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:13:53.661481 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:13:53.661481 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:13:53.661489 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:13:53.662964 (+ 1475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:13:53.663371 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:13:53.663372 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:13:53.663372 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:13:53.663434 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:13:53.663659 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:13:53.663660 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:13:53.663660 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:13:53.663779 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:13:53.663835 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 01:13:53.664107 (+ 272us) server_negotiation.cc:299] Negotiation successful
0926 01:13:53.664201 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260926 01:13:53.664505 19732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:53.655506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39609 (local address 127.0.98.1:60875)
0926 01:13:53.655688 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0926 01:13:53.655724 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 01:13:53.656649 (+ 925us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:13:53.656895 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:13:53.656899 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:13:53.657159 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:13:53.657352 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:13:53.657356 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:13:53.658094 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:13:53.658095 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:13:53.658907 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:13:53.658909 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:13:53.659280 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:13:53.659281 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:13:53.659401 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:53.659898 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:13:53.659906 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:13:53.661386 (+ 1480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:13:53.663042 (+ 1656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:13:53.663051 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:13:53.663054 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:13:53.663301 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:13:53.663489 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:13:53.663489 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:13:53.663490 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:13:53.663606 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:13:53.663831 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:13:53.663839 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:13:53.663986 (+ 147us) client_negotiation.cc:769] Sending connection context
0926 01:13:53.664076 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 01:13:53.664285 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":81,"threads_started":1}
I20260926 01:13:53.665011 19730 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39609
I20260926 01:13:53.665113 19730 heartbeater.cc:464] Registering TS with master...
I20260926 01:13:53.665298 19730 heartbeater.cc:511] Master 127.0.98.62:39609 requested a full tablet report, sending...
I20260926 01:13:53.665907 19539 ts_manager.cc:194] Registered new tserver with Master: 7a02b46540a548249f2cd72d6f0b51c5 (127.0.98.1:44207)
I20260926 01:13:53.666833 19539 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:60875
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384993574762-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384993574762-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 01:13:53.689739 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384993574762-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384993574762-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384993574762-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384993574762-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39609
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384993574762-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:13:53.772238 19735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:13:53.772464 19735 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:13:53.772488 19735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:13:53.773661 19735 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:13:53.773702 19735 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:13:53.773728 19735 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 01:13:53.775276 19735 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384993574762-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384993574762-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384993574762-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384993574762-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39609
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19735
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384993574762-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:13:53.775475 19735 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:13:53.775660 19735 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 01:13:53.780858 19735 server_base.cc:1034] running on GCE node
W20260926 01:13:53.780942 19743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:13:53.780889 19740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:13:53.780921 19741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:13:53.781289 19735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:13:53.781548 19735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:13:53.782691 19735 hybrid_clock.cc:648] HybridClock initialized: now 1790385233782674 us; error 30 us; skew 500 ppm
Sep 26 01:13:53 dist-test-slave-7b22 krb5kdc[19494](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385233, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:13:53.785434 19735 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 01:13:53.785931 19735 webserver.cc:460] Webserver started at http://127.0.98.2:42933/ using document root <none> and password file <none>
I20260926 01:13:53.786144 19735 fs_manager.cc:359] Metadata directory not provided
I20260926 01:13:53.786190 19735 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:13:53.786298 19735 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:13:53.787415 19735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384993574762-392-0/minicluster-data/ts-1/data/instance:
uuid: "f2a22e3914ae45adaac96b70c75e6f63"
format_stamp: "Formatted at 2026-09-26 01:13:53 on dist-test-slave-7b22"
server_key: "8f1ea1bc1377ed90438ce8c9110207e4"
server_key_iv: "66d40529c57aa7175372dba93930da42"
server_key_version: "encryptionkey@0"
I20260926 01:13:53.787751 19735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384993574762-392-0/minicluster-data/ts-1/wal/instance:
uuid: "f2a22e3914ae45adaac96b70c75e6f63"
format_stamp: "Formatted at 2026-09-26 01:13:53 on dist-test-slave-7b22"
server_key: "8f1ea1bc1377ed90438ce8c9110207e4"
server_key_iv: "66d40529c57aa7175372dba93930da42"
server_key_version: "encryptionkey@0"
I20260926 01:13:53.789103 19735 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 01:13:53.789916 19750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:13:53.790089 19735 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:13:53.790179 19735 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384993574762-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384993574762-392-0/minicluster-data/ts-1/wal
uuid: "f2a22e3914ae45adaac96b70c75e6f63"
format_stamp: "Formatted at 2026-09-26 01:13:53 on dist-test-slave-7b22"
server_key: "8f1ea1bc1377ed90438ce8c9110207e4"
server_key_iv: "66d40529c57aa7175372dba93930da42"
server_key_version: "encryptionkey@0"
I20260926 01:13:53.790246 19735 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384993574762-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384993574762-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384993574762-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:13:53.808632 19735 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:13:53.810865 19735 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:13:53.811013 19735 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:13:53.811245 19735 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:13:53.811553 19735 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:13:53.811586 19735 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:13:53.811617 19735 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:13:53.811640 19735 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:13:53.817760 19735 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43091
I20260926 01:13:53.817850 19863 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43091 every 8 connection(s)
I20260926 01:13:53.818152 19735 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384993574762-392-0/minicluster-data/ts-1/data/info.pb
Sep 26 01:13:53 dist-test-slave-7b22 krb5kdc[19494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385233, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:13:53.824889 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 19735
I20260926 01:13:53.824987 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384993574762-392-0/minicluster-data/ts-1/wal/instance
I20260926 01:13:53.825222 392 external_mini_cluster.cc:1428] Setting key a5348b96395dc7ba69a6c2e33b282dce
I20260926 01:13:53.827881 19579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:53.818879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35611 (local address 127.0.98.62:39609)
0926 01:13:53.818924 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 01:13:53.818926 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:13:53.819185 (+ 259us) server_negotiation.cc:407] Connection header received
0926 01:13:53.820071 (+ 886us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:13:53.820073 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:13:53.820085 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:13:53.820164 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:13:53.820938 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:53.821451 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:53.822570 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:53.822825 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:53.822903 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:53.825107 (+ 2204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:13:53.825109 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:13:53.825110 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:13:53.825119 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:13:53.826475 (+ 1356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:13:53.826860 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:13:53.826861 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:13:53.826861 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:13:53.826916 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:13:53.827235 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:13:53.827236 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:13:53.827236 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:13:53.827362 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:13:53.827428 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 01:13:53.827734 (+ 306us) server_negotiation.cc:299] Negotiation successful
0926 01:13:53.827812 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 01:13:53.828158 19866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:53.818831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39609 (local address 127.0.98.2:35611)
0926 01:13:53.818986 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 01:13:53.819028 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 01:13:53.819960 (+ 932us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:13:53.820267 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:13:53.820271 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:13:53.820559 (+ 288us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:13:53.820822 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:13:53.820828 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:13:53.821557 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:13:53.821558 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:13:53.822483 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:13:53.822486 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:13:53.822923 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:13:53.822925 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:13:53.823071 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:53.823551 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:13:53.823560 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:13:53.824977 (+ 1417us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:13:53.826553 (+ 1576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:13:53.826561 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:13:53.826563 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:13:53.826793 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:13:53.826989 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:13:53.826990 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:13:53.826991 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:13:53.827152 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:13:53.827427 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:13:53.827431 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:13:53.827605 (+ 174us) client_negotiation.cc:769] Sending connection context
0926 01:13:53.827716 (+ 111us) client_negotiation.cc:240] Negotiation successful
0926 01:13:53.827896 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":85,"threads_started":1}
I20260926 01:13:53.828594 19864 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39609
I20260926 01:13:53.828701 19864 heartbeater.cc:464] Registering TS with master...
I20260926 01:13:53.828879 19864 heartbeater.cc:511] Master 127.0.98.62:39609 requested a full tablet report, sending...
I20260926 01:13:53.829281 19539 ts_manager.cc:194] Registered new tserver with Master: f2a22e3914ae45adaac96b70c75e6f63 (127.0.98.2:43091)
I20260926 01:13:53.829875 19539 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:35611
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384993574762-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384993574762-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 01:13:53.854918 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384993574762-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384993574762-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384993574762-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384993574762-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39609
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384993574762-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:13:53.934830 19869 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:13:53.935052 19869 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:13:53.935076 19869 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:13:53.936223 19869 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:13:53.936261 19869 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:13:53.936286 19869 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 01:13:53.937783 19869 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384993574762-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384993574762-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384993574762-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384993574762-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39609
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19869
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384993574762-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:13:53.938022 19869 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:13:53.938211 19869 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:13:53.943375 19875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:13:53.943368 19874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:13:53.943368 19877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:13:53.943396 19869 server_base.cc:1034] running on GCE node
I20260926 01:13:53.943790 19869 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:13:53.944031 19869 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:13:53.945174 19869 hybrid_clock.cc:648] HybridClock initialized: now 1790385233945159 us; error 34 us; skew 500 ppm
Sep 26 01:13:53 dist-test-slave-7b22 krb5kdc[19494](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385233, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:13:53.947889 19869 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 01:13:53.948343 19869 webserver.cc:460] Webserver started at http://127.0.98.3:33219/ using document root <none> and password file <none>
I20260926 01:13:53.948546 19869 fs_manager.cc:359] Metadata directory not provided
I20260926 01:13:53.948590 19869 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:13:53.948698 19869 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:13:53.949882 19869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384993574762-392-0/minicluster-data/ts-2/data/instance:
uuid: "313ad3526ded4f91a3d78357bc76b92a"
format_stamp: "Formatted at 2026-09-26 01:13:53 on dist-test-slave-7b22"
server_key: "5b30fe92ba92df617a560054ecc224fe"
server_key_iv: "c088b653a0e8047e02fb6dd9c6570729"
server_key_version: "encryptionkey@0"
I20260926 01:13:53.950284 19869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384993574762-392-0/minicluster-data/ts-2/wal/instance:
uuid: "313ad3526ded4f91a3d78357bc76b92a"
format_stamp: "Formatted at 2026-09-26 01:13:53 on dist-test-slave-7b22"
server_key: "5b30fe92ba92df617a560054ecc224fe"
server_key_iv: "c088b653a0e8047e02fb6dd9c6570729"
server_key_version: "encryptionkey@0"
I20260926 01:13:53.951608 19869 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 01:13:53.952356 19884 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:13:53.952487 19869 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 01:13:53.952569 19869 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384993574762-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384993574762-392-0/minicluster-data/ts-2/wal
uuid: "313ad3526ded4f91a3d78357bc76b92a"
format_stamp: "Formatted at 2026-09-26 01:13:53 on dist-test-slave-7b22"
server_key: "5b30fe92ba92df617a560054ecc224fe"
server_key_iv: "c088b653a0e8047e02fb6dd9c6570729"
server_key_version: "encryptionkey@0"
I20260926 01:13:53.952637 19869 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384993574762-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384993574762-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384993574762-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:13:53.964119 19869 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:13:53.966538 19869 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:13:53.966660 19869 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:13:53.966923 19869 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:13:53.967228 19869 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:13:53.967260 19869 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:13:53.967289 19869 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:13:53.967312 19869 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:13:53.973165 19869 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42625
I20260926 01:13:53.973253 19997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42625 every 8 connection(s)
I20260926 01:13:53.973543 19869 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384993574762-392-0/minicluster-data/ts-2/data/info.pb
I20260926 01:13:53.979837 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 19869
I20260926 01:13:53.979954 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384993574762-392-0/minicluster-data/ts-2/wal/instance
Sep 26 01:13:53 dist-test-slave-7b22 krb5kdc[19494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385233, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:13:53.980118 392 external_mini_cluster.cc:1428] Setting key 711ad4b890b8f54b507c2a7ec6e80ed4
I20260926 01:13:53.983265 19579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:53.974278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34687 (local address 127.0.98.62:39609)
0926 01:13:53.974335 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 01:13:53.974337 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:13:53.974512 (+ 175us) server_negotiation.cc:407] Connection header received
0926 01:13:53.975486 (+ 974us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:13:53.975488 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:13:53.975502 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:13:53.975541 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:13:53.976238 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:53.976859 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:53.978043 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:53.978242 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:53.978299 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:53.980574 (+ 2275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:13:53.980575 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:13:53.980575 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:13:53.980583 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:13:53.981944 (+ 1361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:13:53.982326 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:13:53.982327 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:13:53.982327 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:13:53.982374 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:13:53.982572 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:13:53.982572 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:13:53.982572 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:13:53.982673 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:13:53.982720 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0926 01:13:53.983060 (+ 340us) server_negotiation.cc:299] Negotiation successful
0926 01:13:53.983168 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260926 01:13:53.983327 20000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:53.974261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39609 (local address 127.0.98.3:34687)
0926 01:13:53.974397 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0926 01:13:53.974432 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 01:13:53.975380 (+ 948us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:13:53.975611 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:13:53.975615 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:13:53.975892 (+ 277us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:13:53.976152 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:13:53.976157 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:13:53.976941 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:13:53.976942 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:13:53.977965 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:13:53.977968 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:13:53.978311 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:13:53.978312 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:13:53.978438 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:53.979061 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:13:53.979070 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:13:53.980490 (+ 1420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:13:53.982027 (+ 1537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:13:53.982033 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:13:53.982036 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:13:53.982263 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:13:53.982426 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:13:53.982427 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:13:53.982427 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:13:53.982523 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:13:53.982767 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:13:53.982773 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:13:53.982902 (+ 129us) client_negotiation.cc:769] Sending connection context
0926 01:13:53.983007 (+ 105us) client_negotiation.cc:240] Negotiation successful
0926 01:13:53.983153 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":67,"threads_started":1}
I20260926 01:13:53.983835 19998 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39609
I20260926 01:13:53.983922 19998 heartbeater.cc:464] Registering TS with master...
I20260926 01:13:53.984072 19998 heartbeater.cc:511] Master 127.0.98.62:39609 requested a full tablet report, sending...
I20260926 01:13:53.984404 19539 ts_manager.cc:194] Registered new tserver with Master: 313ad3526ded4f91a3d78357bc76b92a (127.0.98.3:42625)
I20260926 01:13:53.984771 19539 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:34687
I20260926 01:13:53.992100 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 01:13:53.999225 19579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:53.993511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33736 (local address 127.0.98.62:39609)
0926 01:13:53.993581 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0926 01:13:53.993583 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:13:53.993657 (+ 74us) server_negotiation.cc:407] Connection header received
0926 01:13:53.993728 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:13:53.993729 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:13:53.993739 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:13:53.993785 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:13:53.994420 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:53.994908 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:53.995625 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:53.995810 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:53.995864 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:53.997023 (+ 1159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:13:53.997025 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:13:53.997025 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:13:53.997032 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:13:53.998192 (+ 1160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:13:53.998488 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:13:53.998489 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:13:53.998489 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:13:53.998541 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:13:53.998701 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:13:53.998702 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:13:53.998702 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:13:53.998850 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:13:53.998936 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 01:13:53.999055 (+ 119us) server_negotiation.cc:299] Negotiation successful
0926 01:13:53.999133 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 01:13:54.000833 19539 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33736:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:13:54.002439 19539 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:13:54.008594 20012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:54.004742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53624 (local address 127.0.98.2:43091)
0926 01:13:54.004929 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 01:13:54.004933 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 01:13:54.005035 (+ 102us) server_negotiation.cc:407] Connection header received
0926 01:13:54.005079 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:13:54.005081 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:13:54.005096 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:13:54.005163 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:13:54.005402 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:54.006234 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:54.007652 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:54.008202 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:54.008272 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:54.008335 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 01:13:54.008468 (+ 133us) server_negotiation.cc:299] Negotiation successful
0926 01:13:54.008507 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":79,"threads_started":1}
I20260926 01:13:54.008589 20011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:54.004563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43091 (local address 127.0.0.1:53624)
0926 01:13:54.004970 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0926 01:13:54.004978 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 01:13:54.005025 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:13:54.005251 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:13:54.005251 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:13:54.005253 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:13:54.005323 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:13:54.005325 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:13:54.006322 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:13:54.006322 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:13:54.007553 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:13:54.007555 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:13:54.008277 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:13:54.008278 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:13:54.008369 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:54.008382 (+ 13us) client_negotiation.cc:769] Sending connection context
0926 01:13:54.008446 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 01:13:54.008505 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":97,"threads_started":1}
I20260926 01:13:54.008739 20009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:54.004561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44207 (local address 127.0.0.1:59114)
0926 01:13:54.005023 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0926 01:13:54.005033 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 01:13:54.005097 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:13:54.005256 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:13:54.005257 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:13:54.005258 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:13:54.005330 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:13:54.005331 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:13:54.006717 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:13:54.006718 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:13:54.007543 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:13:54.007545 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:13:54.008433 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:13:54.008434 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:13:54.008493 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:54.008498 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 01:13:54.008542 (+ 44us) client_negotiation.cc:240] Negotiation successful
0926 01:13:54.008583 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"mutex_wait_us":23,"thread_start_us":100,"threads_started":1}
I20260926 01:13:54.008852 20010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:54.004561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42625 (local address 127.0.0.1:50126)
0926 01:13:54.004720 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 01:13:54.004799 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0926 01:13:54.004867 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:13:54.005233 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:13:54.005236 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:13:54.005238 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:13:54.005326 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:13:54.005328 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:13:54.006676 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:13:54.006677 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:13:54.007733 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:13:54.007735 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:13:54.008636 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:13:54.008637 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:13:54.008693 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:54.008696 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 01:13:54.008742 (+ 46us) client_negotiation.cc:240] Negotiation successful
0926 01:13:54.008778 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"mutex_wait_us":2,"thread_start_us":45,"threads_started":1}
I20260926 01:13:54.008908 20014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:54.004811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59114 (local address 127.0.98.1:44207)
0926 01:13:54.004978 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 01:13:54.004982 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 01:13:54.005096 (+ 114us) server_negotiation.cc:407] Connection header received
0926 01:13:54.005169 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:13:54.005172 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:13:54.005187 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:13:54.005253 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:13:54.005517 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:54.006622 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:54.007608 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:54.008281 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:54.008345 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:54.008399 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 01:13:54.008630 (+ 231us) server_negotiation.cc:299] Negotiation successful
0926 01:13:54.008778 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":70,"threads_started":1}
I20260926 01:13:54.009130 20013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:54.004800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50126 (local address 127.0.98.3:42625)
0926 01:13:54.004978 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 01:13:54.004982 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 01:13:54.004993 (+ 11us) server_negotiation.cc:407] Connection header received
0926 01:13:54.005034 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:13:54.005037 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:13:54.005053 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:13:54.005123 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:13:54.005404 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:54.006576 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:54.007818 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:54.008486 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:54.008545 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:54.008592 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0926 01:13:54.009012 (+ 420us) server_negotiation.cc:299] Negotiation successful
0926 01:13:54.009055 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":80,"threads_started":1}
I20260926 01:13:54.009579 19664 tablet_service.cc:1467] Processing CreateTablet for tablet adeea1533d574f72a961b3122692a479 (DEFAULT_TABLE table=test-table [id=6e866d121ada40bfa9293cb4606998c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:13:54.009680 19932 tablet_service.cc:1467] Processing CreateTablet for tablet adeea1533d574f72a961b3122692a479 (DEFAULT_TABLE table=test-table [id=6e866d121ada40bfa9293cb4606998c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:13:54.009680 19798 tablet_service.cc:1467] Processing CreateTablet for tablet adeea1533d574f72a961b3122692a479 (DEFAULT_TABLE table=test-table [id=6e866d121ada40bfa9293cb4606998c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:13:54.009970 19664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adeea1533d574f72a961b3122692a479. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:13:54.010023 19932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adeea1533d574f72a961b3122692a479. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:13:54.010023 19798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adeea1533d574f72a961b3122692a479. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:13:54.013530 20017 tablet_bootstrap.cc:492] T adeea1533d574f72a961b3122692a479 P 313ad3526ded4f91a3d78357bc76b92a: Bootstrap starting.
I20260926 01:13:54.013552 20015 tablet_bootstrap.cc:492] T adeea1533d574f72a961b3122692a479 P 7a02b46540a548249f2cd72d6f0b51c5: Bootstrap starting.
I20260926 01:13:54.013989 20016 tablet_bootstrap.cc:492] T adeea1533d574f72a961b3122692a479 P f2a22e3914ae45adaac96b70c75e6f63: Bootstrap starting.
I20260926 01:13:54.014359 20015 tablet_bootstrap.cc:654] T adeea1533d574f72a961b3122692a479 P 7a02b46540a548249f2cd72d6f0b51c5: Neither blocks nor log segments found. Creating new log.
I20260926 01:13:54.014364 20017 tablet_bootstrap.cc:654] T adeea1533d574f72a961b3122692a479 P 313ad3526ded4f91a3d78357bc76b92a: Neither blocks nor log segments found. Creating new log.
I20260926 01:13:54.014897 20015 log.cc:824] T adeea1533d574f72a961b3122692a479 P 7a02b46540a548249f2cd72d6f0b51c5: Log is configured to *not* fsync() on all Append() calls
I20260926 01:13:54.014897 20017 log.cc:824] T adeea1533d574f72a961b3122692a479 P 313ad3526ded4f91a3d78357bc76b92a: Log is configured to *not* fsync() on all Append() calls
I20260926 01:13:54.014945 20016 tablet_bootstrap.cc:654] T adeea1533d574f72a961b3122692a479 P f2a22e3914ae45adaac96b70c75e6f63: Neither blocks nor log segments found. Creating new log.
I20260926 01:13:54.015496 20016 log.cc:824] T adeea1533d574f72a961b3122692a479 P f2a22e3914ae45adaac96b70c75e6f63: Log is configured to *not* fsync() on all Append() calls
I20260926 01:13:54.016260 20016 tablet_bootstrap.cc:492] T adeea1533d574f72a961b3122692a479 P f2a22e3914ae45adaac96b70c75e6f63: No bootstrap required, opened a new log
I20260926 01:13:54.016340 20016 ts_tablet_manager.cc:1397] T adeea1533d574f72a961b3122692a479 P f2a22e3914ae45adaac96b70c75e6f63: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 01:13:54.017716 20016 raft_consensus.cc:348] T adeea1533d574f72a961b3122692a479 P f2a22e3914ae45adaac96b70c75e6f63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a02b46540a548249f2cd72d6f0b51c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44207 } } peers { permanent_uuid: "313ad3526ded4f91a3d78357bc76b92a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42625 } } peers { permanent_uuid: "f2a22e3914ae45adaac96b70c75e6f63" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43091 } }
I20260926 01:13:54.017925 20016 raft_consensus.cc:374] T adeea1533d574f72a961b3122692a479 P f2a22e3914ae45adaac96b70c75e6f63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:13:54.017966 20016 raft_consensus.cc:729] T adeea1533d574f72a961b3122692a479 P f2a22e3914ae45adaac96b70c75e6f63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2a22e3914ae45adaac96b70c75e6f63, State: Initialized, Role: FOLLOWER
I20260926 01:13:54.018085 20016 consensus_queue.cc:260] T adeea1533d574f72a961b3122692a479 P f2a22e3914ae45adaac96b70c75e6f63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a02b46540a548249f2cd72d6f0b51c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44207 } } peers { permanent_uuid: "313ad3526ded4f91a3d78357bc76b92a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42625 } } peers { permanent_uuid: "f2a22e3914ae45adaac96b70c75e6f63" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43091 } }
I20260926 01:13:54.018356 20016 ts_tablet_manager.cc:1428] T adeea1533d574f72a961b3122692a479 P f2a22e3914ae45adaac96b70c75e6f63: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 01:13:54.018792 20017 tablet_bootstrap.cc:492] T adeea1533d574f72a961b3122692a479 P 313ad3526ded4f91a3d78357bc76b92a: No bootstrap required, opened a new log
I20260926 01:13:54.018862 20017 ts_tablet_manager.cc:1397] T adeea1533d574f72a961b3122692a479 P 313ad3526ded4f91a3d78357bc76b92a: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260926 01:13:54.019032 20015 tablet_bootstrap.cc:492] T adeea1533d574f72a961b3122692a479 P 7a02b46540a548249f2cd72d6f0b51c5: No bootstrap required, opened a new log
I20260926 01:13:54.019117 20015 ts_tablet_manager.cc:1397] T adeea1533d574f72a961b3122692a479 P 7a02b46540a548249f2cd72d6f0b51c5: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20260926 01:13:54.020731 20017 raft_consensus.cc:348] T adeea1533d574f72a961b3122692a479 P 313ad3526ded4f91a3d78357bc76b92a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a02b46540a548249f2cd72d6f0b51c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44207 } } peers { permanent_uuid: "313ad3526ded4f91a3d78357bc76b92a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42625 } } peers { permanent_uuid: "f2a22e3914ae45adaac96b70c75e6f63" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43091 } }
I20260926 01:13:54.020918 20017 raft_consensus.cc:374] T adeea1533d574f72a961b3122692a479 P 313ad3526ded4f91a3d78357bc76b92a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:13:54.020946 20017 raft_consensus.cc:729] T adeea1533d574f72a961b3122692a479 P 313ad3526ded4f91a3d78357bc76b92a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 313ad3526ded4f91a3d78357bc76b92a, State: Initialized, Role: FOLLOWER
I20260926 01:13:54.021117 19864 heartbeater.cc:503] Master 127.0.98.62:39609 was elected leader, sending a full tablet report...
I20260926 01:13:54.021415 20017 consensus_queue.cc:260] T adeea1533d574f72a961b3122692a479 P 313ad3526ded4f91a3d78357bc76b92a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a02b46540a548249f2cd72d6f0b51c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44207 } } peers { permanent_uuid: "313ad3526ded4f91a3d78357bc76b92a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42625 } } peers { permanent_uuid: "f2a22e3914ae45adaac96b70c75e6f63" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43091 } }
I20260926 01:13:54.021723 20017 ts_tablet_manager.cc:1428] T adeea1533d574f72a961b3122692a479 P 313ad3526ded4f91a3d78357bc76b92a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 01:13:54.021867 19998 heartbeater.cc:503] Master 127.0.98.62:39609 was elected leader, sending a full tablet report...
I20260926 01:13:54.022897 20015 raft_consensus.cc:348] T adeea1533d574f72a961b3122692a479 P 7a02b46540a548249f2cd72d6f0b51c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a02b46540a548249f2cd72d6f0b51c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44207 } } peers { permanent_uuid: "313ad3526ded4f91a3d78357bc76b92a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42625 } } peers { permanent_uuid: "f2a22e3914ae45adaac96b70c75e6f63" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43091 } }
I20260926 01:13:54.023139 20015 raft_consensus.cc:374] T adeea1533d574f72a961b3122692a479 P 7a02b46540a548249f2cd72d6f0b51c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:13:54.023285 20015 raft_consensus.cc:729] T adeea1533d574f72a961b3122692a479 P 7a02b46540a548249f2cd72d6f0b51c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a02b46540a548249f2cd72d6f0b51c5, State: Initialized, Role: FOLLOWER
I20260926 01:13:54.023449 20015 consensus_queue.cc:260] T adeea1533d574f72a961b3122692a479 P 7a02b46540a548249f2cd72d6f0b51c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a02b46540a548249f2cd72d6f0b51c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44207 } } peers { permanent_uuid: "313ad3526ded4f91a3d78357bc76b92a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42625 } } peers { permanent_uuid: "f2a22e3914ae45adaac96b70c75e6f63" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43091 } }
I20260926 01:13:54.023986 20015 ts_tablet_manager.cc:1428] T adeea1533d574f72a961b3122692a479 P 7a02b46540a548249f2cd72d6f0b51c5: Time spent starting tablet: real 0.005s user 0.003s sys 0.001s
I20260926 01:13:54.025123 19730 heartbeater.cc:503] Master 127.0.98.62:39609 was elected leader, sending a full tablet report...
I20260926 01:13:54.041272 20022 raft_consensus.cc:482] T adeea1533d574f72a961b3122692a479 P 313ad3526ded4f91a3d78357bc76b92a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:13:54.041401 20022 raft_consensus.cc:504] T adeea1533d574f72a961b3122692a479 P 313ad3526ded4f91a3d78357bc76b92a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a02b46540a548249f2cd72d6f0b51c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44207 } } peers { permanent_uuid: "313ad3526ded4f91a3d78357bc76b92a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42625 } } peers { permanent_uuid: "f2a22e3914ae45adaac96b70c75e6f63" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43091 } }
I20260926 01:13:54.041730 20022 leader_election.cc:288] T adeea1533d574f72a961b3122692a479 P 313ad3526ded4f91a3d78357bc76b92a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a02b46540a548249f2cd72d6f0b51c5 (127.0.98.1:44207), f2a22e3914ae45adaac96b70c75e6f63 (127.0.98.2:43091)
I20260926 01:13:54.044672 20000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:54.041929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44207 (local address 127.0.98.3:39361)
0926 01:13:54.041972 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0926 01:13:54.041983 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 01:13:54.042025 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:13:54.042185 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:13:54.042186 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:13:54.042189 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:13:54.042262 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:13:54.042263 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:13:54.043001 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:13:54.043002 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:13:54.043868 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:13:54.043871 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:13:54.044425 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:13:54.044426 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:13:54.044505 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:54.044509 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 01:13:54.044559 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 01:13:54.044601 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21,"mutex_wait_us":27}
I20260926 01:13:54.044756 20014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:54.041999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39361 (local address 127.0.98.1:44207)
0926 01:13:54.042063 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 01:13:54.042065 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:13:54.042073 (+ 8us) server_negotiation.cc:407] Connection header received
0926 01:13:54.042098 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:13:54.042099 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:13:54.042108 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:13:54.042165 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:13:54.042357 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:54.042920 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:54.043953 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:54.044352 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:54.044402 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:54.044411 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 01:13:54.044595 (+ 184us) server_negotiation.cc:299] Negotiation successful
0926 01:13:54.044643 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 01:13:54.044857 20024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:54.041929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43091 (local address 127.0.98.3:52447)
0926 01:13:54.042263 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0926 01:13:54.042272 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 01:13:54.042326 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:13:54.042449 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:13:54.042449 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:13:54.042451 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:13:54.042518 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:13:54.042520 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:13:54.043177 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:13:54.043178 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:13:54.044011 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:13:54.044013 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:13:54.044597 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:13:54.044598 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:13:54.044673 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:54.044677 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 01:13:54.044741 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 01:13:54.044773 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"mutex_wait_us":142,"thread_start_us":119,"threads_started":1}
I20260926 01:13:54.045055 19684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adeea1533d574f72a961b3122692a479" candidate_uuid: "313ad3526ded4f91a3d78357bc76b92a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a02b46540a548249f2cd72d6f0b51c5" is_pre_election: true
I20260926 01:13:54.045178 20012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:54.042004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52447 (local address 127.0.98.2:43091)
0926 01:13:54.042066 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 01:13:54.042068 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:13:54.042334 (+ 266us) server_negotiation.cc:407] Connection header received
0926 01:13:54.042378 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:13:54.042379 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:13:54.042391 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:13:54.042433 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:13:54.042591 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:54.043093 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:54.044093 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:54.044494 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:54.044565 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:54.044575 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 01:13:54.045070 (+ 495us) server_negotiation.cc:299] Negotiation successful
0926 01:13:54.045110 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 01:13:54.045212 19684 raft_consensus.cc:2454] T adeea1533d574f72a961b3122692a479 P 7a02b46540a548249f2cd72d6f0b51c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 313ad3526ded4f91a3d78357bc76b92a in term 0.
I20260926 01:13:54.045447 19888 leader_election.cc:302] T adeea1533d574f72a961b3122692a479 P 313ad3526ded4f91a3d78357bc76b92a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 313ad3526ded4f91a3d78357bc76b92a, 7a02b46540a548249f2cd72d6f0b51c5; no voters:
I20260926 01:13:54.045514 19818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adeea1533d574f72a961b3122692a479" candidate_uuid: "313ad3526ded4f91a3d78357bc76b92a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2a22e3914ae45adaac96b70c75e6f63" is_pre_election: true
I20260926 01:13:54.045585 20022 raft_consensus.cc:2781] T adeea1533d574f72a961b3122692a479 P 313ad3526ded4f91a3d78357bc76b92a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:13:54.045634 20022 raft_consensus.cc:482] T adeea1533d574f72a961b3122692a479 P 313ad3526ded4f91a3d78357bc76b92a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:13:54.045653 20022 raft_consensus.cc:3037] T adeea1533d574f72a961b3122692a479 P 313ad3526ded4f91a3d78357bc76b92a [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:13:54.045651 19818 raft_consensus.cc:2454] T adeea1533d574f72a961b3122692a479 P f2a22e3914ae45adaac96b70c75e6f63 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 313ad3526ded4f91a3d78357bc76b92a in term 0.
I20260926 01:13:54.046379 20022 raft_consensus.cc:504] T adeea1533d574f72a961b3122692a479 P 313ad3526ded4f91a3d78357bc76b92a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a02b46540a548249f2cd72d6f0b51c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44207 } } peers { permanent_uuid: "313ad3526ded4f91a3d78357bc76b92a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42625 } } peers { permanent_uuid: "f2a22e3914ae45adaac96b70c75e6f63" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43091 } }
I20260926 01:13:54.046497 20022 leader_election.cc:288] T adeea1533d574f72a961b3122692a479 P 313ad3526ded4f91a3d78357bc76b92a [CANDIDATE]: Term 1 election: Requested vote from peers 7a02b46540a548249f2cd72d6f0b51c5 (127.0.98.1:44207), f2a22e3914ae45adaac96b70c75e6f63 (127.0.98.2:43091)
I20260926 01:13:54.046638 19818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adeea1533d574f72a961b3122692a479" candidate_uuid: "313ad3526ded4f91a3d78357bc76b92a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2a22e3914ae45adaac96b70c75e6f63"
I20260926 01:13:54.046638 19684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adeea1533d574f72a961b3122692a479" candidate_uuid: "313ad3526ded4f91a3d78357bc76b92a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a02b46540a548249f2cd72d6f0b51c5"
I20260926 01:13:54.046725 19684 raft_consensus.cc:3037] T adeea1533d574f72a961b3122692a479 P 7a02b46540a548249f2cd72d6f0b51c5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:13:54.046725 19818 raft_consensus.cc:3037] T adeea1533d574f72a961b3122692a479 P f2a22e3914ae45adaac96b70c75e6f63 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:13:54.047600 19684 raft_consensus.cc:2454] T adeea1533d574f72a961b3122692a479 P 7a02b46540a548249f2cd72d6f0b51c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 313ad3526ded4f91a3d78357bc76b92a in term 1.
I20260926 01:13:54.047600 19818 raft_consensus.cc:2454] T adeea1533d574f72a961b3122692a479 P f2a22e3914ae45adaac96b70c75e6f63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 313ad3526ded4f91a3d78357bc76b92a in term 1.
I20260926 01:13:54.047791 19888 leader_election.cc:302] T adeea1533d574f72a961b3122692a479 P 313ad3526ded4f91a3d78357bc76b92a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 313ad3526ded4f91a3d78357bc76b92a, 7a02b46540a548249f2cd72d6f0b51c5; no voters:
I20260926 01:13:54.047912 20022 raft_consensus.cc:2781] T adeea1533d574f72a961b3122692a479 P 313ad3526ded4f91a3d78357bc76b92a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:13:54.048082 20022 raft_consensus.cc:686] T adeea1533d574f72a961b3122692a479 P 313ad3526ded4f91a3d78357bc76b92a [term 1 LEADER]: Becoming Leader. State: Replica: 313ad3526ded4f91a3d78357bc76b92a, State: Running, Role: LEADER
I20260926 01:13:54.048175 20022 consensus_queue.cc:237] T adeea1533d574f72a961b3122692a479 P 313ad3526ded4f91a3d78357bc76b92a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a02b46540a548249f2cd72d6f0b51c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44207 } } peers { permanent_uuid: "313ad3526ded4f91a3d78357bc76b92a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42625 } } peers { permanent_uuid: "f2a22e3914ae45adaac96b70c75e6f63" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43091 } }
I20260926 01:13:54.049008 19539 catalog_manager.cc:5495] T adeea1533d574f72a961b3122692a479 P 313ad3526ded4f91a3d78357bc76b92a reported cstate change: term changed from 0 to 1, leader changed from <none> to 313ad3526ded4f91a3d78357bc76b92a (127.0.98.3). New cstate: current_term: 1 leader_uuid: "313ad3526ded4f91a3d78357bc76b92a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a02b46540a548249f2cd72d6f0b51c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44207 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "313ad3526ded4f91a3d78357bc76b92a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42625 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f2a22e3914ae45adaac96b70c75e6f63" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43091 } health_report { overall_health: UNKNOWN } } }
I20260926 01:13:54.059428 20013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:54.057305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50134 (local address 127.0.98.3:42625)
0926 01:13:54.057354 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 01:13:54.057356 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:13:54.057366 (+ 10us) server_negotiation.cc:407] Connection header received
0926 01:13:54.057412 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:13:54.057413 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:13:54.057428 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:13:54.057478 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:13:54.057598 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:54.058124 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:54.058855 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:54.059032 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:54.059071 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:54.059175 (+ 104us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:13:54.059285 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:13:54.059339 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 01:13:54.059352 (+ 13us) server_negotiation.cc:299] Negotiation successful
0926 01:13:54.059369 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 01:13:54.061224 19684 raft_consensus.cc:1261] T adeea1533d574f72a961b3122692a479 P 7a02b46540a548249f2cd72d6f0b51c5 [term 1 FOLLOWER]: Refusing update from remote peer 313ad3526ded4f91a3d78357bc76b92a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:13:54.061224 19818 raft_consensus.cc:1261] T adeea1533d574f72a961b3122692a479 P f2a22e3914ae45adaac96b70c75e6f63 [term 1 FOLLOWER]: Refusing update from remote peer 313ad3526ded4f91a3d78357bc76b92a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:13:54.061570 20025 consensus_queue.cc:1035] T adeea1533d574f72a961b3122692a479 P 313ad3526ded4f91a3d78357bc76b92a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2a22e3914ae45adaac96b70c75e6f63" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:13:54.061690 20022 consensus_queue.cc:1035] T adeea1533d574f72a961b3122692a479 P 313ad3526ded4f91a3d78357bc76b92a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a02b46540a548249f2cd72d6f0b51c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:13:54.063872 20028 mvcc.cc:205] Tried to move back new op lower bound from 7333417918712176640 to 7333417918662033408. Current Snapshot: MvccSnapshot[applied={T|T < 7333417918712176640}]
I20260926 01:13:54.066028 20030 mvcc.cc:205] Tried to move back new op lower bound from 7333417918712176640 to 7333417918662033408. Current Snapshot: MvccSnapshot[applied={T|T < 7333417918712176640}]
I20260926 01:13:54.066629 19539 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33736:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:13:54.066715 19539 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33736:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:13:54.067376 19539 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c43b423c7517487bac9b8fbd3afad894: Sending DeleteTablet for 3 replicas of tablet adeea1533d574f72a961b3122692a479
I20260926 01:13:54.067667 19664 tablet_service.cc:1514] Processing DeleteTablet for tablet adeea1533d574f72a961b3122692a479 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:13:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59114
I20260926 01:13:54.067703 19798 tablet_service.cc:1514] Processing DeleteTablet for tablet adeea1533d574f72a961b3122692a479 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:13:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53624
I20260926 01:13:54.067832 19932 tablet_service.cc:1514] Processing DeleteTablet for tablet adeea1533d574f72a961b3122692a479 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:13:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50126
I20260926 01:13:54.068037 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 19601
I20260926 01:13:54.069000 20040 tablet_replica.cc:330] T adeea1533d574f72a961b3122692a479 P 313ad3526ded4f91a3d78357bc76b92a: stopping tablet replica
I20260926 01:13:54.069152 20040 raft_consensus.cc:2229] T adeea1533d574f72a961b3122692a479 P 313ad3526ded4f91a3d78357bc76b92a [term 1 LEADER]: Raft consensus shutting down.
I20260926 01:13:54.069322 20040 raft_consensus.cc:2258] T adeea1533d574f72a961b3122692a479 P 313ad3526ded4f91a3d78357bc76b92a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:13:54.069842 20040 ts_tablet_manager.cc:1905] T adeea1533d574f72a961b3122692a479 P 313ad3526ded4f91a3d78357bc76b92a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:13:54.070552 20039 tablet_replica.cc:330] T adeea1533d574f72a961b3122692a479 P f2a22e3914ae45adaac96b70c75e6f63: stopping tablet replica
I20260926 01:13:54.070657 20039 raft_consensus.cc:2229] T adeea1533d574f72a961b3122692a479 P f2a22e3914ae45adaac96b70c75e6f63 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:13:54.070757 20039 raft_consensus.cc:2258] T adeea1533d574f72a961b3122692a479 P f2a22e3914ae45adaac96b70c75e6f63 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:13:54.071168 20039 ts_tablet_manager.cc:1905] T adeea1533d574f72a961b3122692a479 P f2a22e3914ae45adaac96b70c75e6f63: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:13:54.071810 20040 ts_tablet_manager.cc:1918] T adeea1533d574f72a961b3122692a479 P 313ad3526ded4f91a3d78357bc76b92a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:13:54.071928 20040 log.cc:1196] T adeea1533d574f72a961b3122692a479 P 313ad3526ded4f91a3d78357bc76b92a: Deleting WAL directory at /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384993574762-392-0/minicluster-data/ts-2/wal/wals/adeea1533d574f72a961b3122692a479
I20260926 01:13:54.072266 20040 ts_tablet_manager.cc:1939] T adeea1533d574f72a961b3122692a479 P 313ad3526ded4f91a3d78357bc76b92a: Deleting consensus metadata
I20260926 01:13:54.072652 20039 ts_tablet_manager.cc:1918] T adeea1533d574f72a961b3122692a479 P f2a22e3914ae45adaac96b70c75e6f63: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:13:54.072715 20039 log.cc:1196] T adeea1533d574f72a961b3122692a479 P f2a22e3914ae45adaac96b70c75e6f63: Deleting WAL directory at /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384993574762-392-0/minicluster-data/ts-1/wal/wals/adeea1533d574f72a961b3122692a479
I20260926 01:13:54.072758 19523 catalog_manager.cc:4841] TS 313ad3526ded4f91a3d78357bc76b92a (127.0.98.3:42625): tablet adeea1533d574f72a961b3122692a479 (table test-table [id=6e866d121ada40bfa9293cb4606998c2]) successfully deleted
I20260926 01:13:54.072937 20039 ts_tablet_manager.cc:1939] T adeea1533d574f72a961b3122692a479 P f2a22e3914ae45adaac96b70c75e6f63: Deleting consensus metadata
I20260926 01:13:54.073381 19525 catalog_manager.cc:4841] TS f2a22e3914ae45adaac96b70c75e6f63 (127.0.98.2:43091): tablet adeea1533d574f72a961b3122692a479 (table test-table [id=6e866d121ada40bfa9293cb4606998c2]) successfully deleted
W20260926 01:13:54.073653 19526 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44207 (error 108)
I20260926 01:13:54.073868 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 19735
I20260926 01:13:54.074473 20010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:54.074051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44207 (local address 127.0.0.1:59122)
0926 01:13:54.074404 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0926 01:13:54.074440 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334}
W20260926 01:13:54.074755 19526 catalog_manager.cc:4584] TS 7a02b46540a548249f2cd72d6f0b51c5 (127.0.98.1:44207): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet adeea1533d574f72a961b3122692a479: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44207: connect: Connection refused (error 111)
I20260926 01:13:54.078904 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 19869
I20260926 01:13:54.083366 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 19507
2026-09-26T01:13:54Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (1818 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-26T01:13:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:13:54Z Disabled control of system clock
I20260926 01:13:54.109515 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384993574762-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384993574762-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384993574762-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37517
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:44431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37517
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 01:13:54.189436 20045 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:13:54.189708 20045 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:13:54.189741 20045 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:13:54.191082 20045 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:13:54.191150 20045 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:13:54.191166 20045 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:13:54.191184 20045 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:13:54.191200 20045 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:13:54.192714 20045 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384993574762-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384993574762-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37517
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37517
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20045
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384993574762-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:13:54.193003 20045 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:13:54.193302 20045 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 01:13:54.198675 20045 server_base.cc:1034] running on GCE node
W20260926 01:13:54.198724 20053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:13:54.198653 20050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:13:54.198634 20051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:13:54.199057 20045 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:13:54.199345 20045 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:13:54.200506 20045 hybrid_clock.cc:648] HybridClock initialized: now 1790385234200487 us; error 29 us; skew 500 ppm
I20260926 01:13:54.201788 20045 webserver.cc:460] Webserver started at http://127.0.98.62:35579/ using document root <none> and password file <none>
I20260926 01:13:54.202045 20045 fs_manager.cc:359] Metadata directory not provided
I20260926 01:13:54.202097 20045 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:13:54.202246 20045 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:13:54.203313 20045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384993574762-392-0/minicluster-data/master-0/data/instance:
uuid: "4ae3adedbc40426f86292abf1ac76001"
format_stamp: "Formatted at 2026-09-26 01:13:54 on dist-test-slave-7b22"
server_key: "7cc5466d71815c98b67274b2790ec02f"
server_key_iv: "eb16b299b9804885eb39d92767e4efb1"
server_key_version: "encryptionkey@0"
I20260926 01:13:54.203641 20045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384993574762-392-0/minicluster-data/master-0/wal/instance:
uuid: "4ae3adedbc40426f86292abf1ac76001"
format_stamp: "Formatted at 2026-09-26 01:13:54 on dist-test-slave-7b22"
server_key: "7cc5466d71815c98b67274b2790ec02f"
server_key_iv: "eb16b299b9804885eb39d92767e4efb1"
server_key_version: "encryptionkey@0"
I20260926 01:13:54.205016 20045 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 01:13:54.205888 20059 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:13:54.206074 20045 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 01:13:54.206182 20045 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384993574762-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384993574762-392-0/minicluster-data/master-0/wal
uuid: "4ae3adedbc40426f86292abf1ac76001"
format_stamp: "Formatted at 2026-09-26 01:13:54 on dist-test-slave-7b22"
server_key: "7cc5466d71815c98b67274b2790ec02f"
server_key_iv: "eb16b299b9804885eb39d92767e4efb1"
server_key_version: "encryptionkey@0"
I20260926 01:13:54.206262 20045 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384993574762-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384993574762-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384993574762-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:13:54.222235 20045 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:13:54.222558 20045 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:13:54.222674 20045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:13:54.226369 20045 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37517
I20260926 01:13:54.226401 20111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37517 every 8 connection(s)
I20260926 01:13:54.226737 20045 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
I20260926 01:13:54.227716 20112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:13:54.230103 20112 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ae3adedbc40426f86292abf1ac76001: Bootstrap starting.
I20260926 01:13:54.230666 20112 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4ae3adedbc40426f86292abf1ac76001: Neither blocks nor log segments found. Creating new log.
I20260926 01:13:54.230903 20112 log.cc:824] T 00000000000000000000000000000000 P 4ae3adedbc40426f86292abf1ac76001: Log is configured to *not* fsync() on all Append() calls
I20260926 01:13:54.231683 20112 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ae3adedbc40426f86292abf1ac76001: No bootstrap required, opened a new log
I20260926 01:13:54.233309 20112 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4ae3adedbc40426f86292abf1ac76001 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ae3adedbc40426f86292abf1ac76001" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37517 } }
I20260926 01:13:54.233457 20112 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4ae3adedbc40426f86292abf1ac76001 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:13:54.233480 20112 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4ae3adedbc40426f86292abf1ac76001 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ae3adedbc40426f86292abf1ac76001, State: Initialized, Role: FOLLOWER
I20260926 01:13:54.233598 20112 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4ae3adedbc40426f86292abf1ac76001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ae3adedbc40426f86292abf1ac76001" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37517 } }
I20260926 01:13:54.233686 20112 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4ae3adedbc40426f86292abf1ac76001 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:13:54.233711 20112 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4ae3adedbc40426f86292abf1ac76001 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:13:54.233743 20112 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4ae3adedbc40426f86292abf1ac76001 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:13:54.234243 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 20045
I20260926 01:13:54.234298 20112 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4ae3adedbc40426f86292abf1ac76001 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ae3adedbc40426f86292abf1ac76001" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37517 } }
I20260926 01:13:54.234351 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384993574762-392-0/minicluster-data/master-0/wal/instance
I20260926 01:13:54.234383 20112 leader_election.cc:302] T 00000000000000000000000000000000 P 4ae3adedbc40426f86292abf1ac76001 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4ae3adedbc40426f86292abf1ac76001; no voters:
I20260926 01:13:54.234534 392 external_mini_cluster.cc:1428] Setting key 56ef6c475bab76b29c585e985324ea05
I20260926 01:13:54.234555 20112 leader_election.cc:288] T 00000000000000000000000000000000 P 4ae3adedbc40426f86292abf1ac76001 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:13:54.234596 20115 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4ae3adedbc40426f86292abf1ac76001 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:13:54.234712 20115 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4ae3adedbc40426f86292abf1ac76001 [term 1 LEADER]: Becoming Leader. State: Replica: 4ae3adedbc40426f86292abf1ac76001, State: Running, Role: LEADER
I20260926 01:13:54.234807 20115 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4ae3adedbc40426f86292abf1ac76001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ae3adedbc40426f86292abf1ac76001" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37517 } }
I20260926 01:13:54.235224 20112 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4ae3adedbc40426f86292abf1ac76001 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:13:54.235312 20117 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ae3adedbc40426f86292abf1ac76001 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4ae3adedbc40426f86292abf1ac76001" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ae3adedbc40426f86292abf1ac76001" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37517 } } }
I20260926 01:13:54.235399 20117 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ae3adedbc40426f86292abf1ac76001 [sys.catalog]: This master's current role is: LEADER
I20260926 01:13:54.235534 20116 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ae3adedbc40426f86292abf1ac76001 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4ae3adedbc40426f86292abf1ac76001. Latest consensus state: current_term: 1 leader_uuid: "4ae3adedbc40426f86292abf1ac76001" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ae3adedbc40426f86292abf1ac76001" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37517 } } }
I20260926 01:13:54.235591 20116 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ae3adedbc40426f86292abf1ac76001 [sys.catalog]: This master's current role is: LEADER
I20260926 01:13:54.235704 20121 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:13:54.236423 20121 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:13:54.238600 20121 catalog_manager.cc:1347] Generated new cluster ID: 2ca759608964440db8141a965824a124
I20260926 01:13:54.238662 20121 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 01:13:54.240633 20120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:54.235024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60536 (local address 127.0.98.62:37517)
0926 01:13:54.235244 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0926 01:13:54.235254 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 01:13:54.235486 (+ 232us) server_negotiation.cc:407] Connection header received
0926 01:13:54.235608 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:13:54.235616 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:13:54.235696 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:13:54.235841 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:13:54.236234 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:54.237593 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:54.238549 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:54.239121 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:54.239235 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:54.239962 (+ 727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:13:54.239966 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:13:54.239966 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:13:54.240165 (+ 199us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:13:54.240205 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:13:54.240219 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:13:54.240281 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 01:13:54.240355 (+ 74us) server_negotiation.cc:299] Negotiation successful
0926 01:13:54.240431 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":80,"threads_started":1}
I20260926 01:13:54.242628 20121 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:13:54.243307 20121 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:13:54.251161 20121 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4ae3adedbc40426f86292abf1ac76001: Generated new TSK 0
I20260926 01:13:54.251375 20121 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 01:13:54.255919 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384993574762-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384993574762-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384993574762-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384993574762-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37517
--builtin_ntp_servers=127.0.98.20:44431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 01:13:54.336901 20136 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:13:54.337118 20136 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:13:54.337149 20136 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:13:54.338378 20136 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:13:54.338423 20136 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:13:54.338457 20136 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 01:13:54.339839 20136 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384993574762-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384993574762-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384993574762-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37517
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20136
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384993574762-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:13:54.340081 20136 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:13:54.340364 20136 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:13:54.345337 20142 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:13:54.345376 20141 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:13:54.345625 20136 server_base.cc:1034] running on GCE node
W20260926 01:13:54.345732 20144 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:13:54.345957 20136 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:13:54.346225 20136 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:13:54.347440 20136 hybrid_clock.cc:648] HybridClock initialized: now 1790385234347420 us; error 32 us; skew 500 ppm
I20260926 01:13:54.348907 20136 webserver.cc:460] Webserver started at http://127.0.98.1:37177/ using document root <none> and password file <none>
I20260926 01:13:54.349165 20136 fs_manager.cc:359] Metadata directory not provided
I20260926 01:13:54.349218 20136 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:13:54.349359 20136 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:13:54.350548 20136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384993574762-392-0/minicluster-data/ts-0/data/instance:
uuid: "5f581ddfbed04dd3b39112156731886d"
format_stamp: "Formatted at 2026-09-26 01:13:54 on dist-test-slave-7b22"
server_key: "29cdaee3223b5dc4b61cabfc7b18ecc3"
server_key_iv: "024c5b8e07f6ea42ec163c435eefa13d"
server_key_version: "encryptionkey@0"
I20260926 01:13:54.350914 20136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384993574762-392-0/minicluster-data/ts-0/wal/instance:
uuid: "5f581ddfbed04dd3b39112156731886d"
format_stamp: "Formatted at 2026-09-26 01:13:54 on dist-test-slave-7b22"
server_key: "29cdaee3223b5dc4b61cabfc7b18ecc3"
server_key_iv: "024c5b8e07f6ea42ec163c435eefa13d"
server_key_version: "encryptionkey@0"
I20260926 01:13:54.352334 20136 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 01:13:54.353159 20150 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:13:54.353314 20136 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 01:13:54.353390 20136 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384993574762-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384993574762-392-0/minicluster-data/ts-0/wal
uuid: "5f581ddfbed04dd3b39112156731886d"
format_stamp: "Formatted at 2026-09-26 01:13:54 on dist-test-slave-7b22"
server_key: "29cdaee3223b5dc4b61cabfc7b18ecc3"
server_key_iv: "024c5b8e07f6ea42ec163c435eefa13d"
server_key_version: "encryptionkey@0"
I20260926 01:13:54.353459 20136 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384993574762-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384993574762-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384993574762-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:13:54.375236 20136 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:13:54.375556 20136 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:13:54.375701 20136 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:13:54.375921 20136 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:13:54.376248 20136 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:13:54.376282 20136 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:13:54.376312 20136 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:13:54.376335 20136 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:13:54.382335 20136 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37089
I20260926 01:13:54.382385 20263 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37089 every 8 connection(s)
I20260926 01:13:54.382715 20136 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384993574762-392-0/minicluster-data/ts-0/data/info.pb
I20260926 01:13:54.388238 20120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:54.383421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35727 (local address 127.0.98.62:37517)
0926 01:13:54.383469 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 01:13:54.383471 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:13:54.383756 (+ 285us) server_negotiation.cc:407] Connection header received
0926 01:13:54.384344 (+ 588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:13:54.384346 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:13:54.384361 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:13:54.384427 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:13:54.385193 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:54.385715 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:54.386791 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:54.386984 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:54.387048 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:54.387888 (+ 840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:13:54.387890 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:13:54.387890 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:13:54.387896 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:13:54.387911 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:13:54.387915 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:13:54.387967 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 01:13:54.388133 (+ 166us) server_negotiation.cc:299] Negotiation successful
0926 01:13:54.388166 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 01:13:54.388404 20266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:54.383401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37517 (local address 127.0.98.1:35727)
0926 01:13:54.383578 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 01:13:54.383680 (+ 102us) client_negotiation.cc:174] Beginning negotiation
0926 01:13:54.384153 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:13:54.384527 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:13:54.384531 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:13:54.384540 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:13:54.385095 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:13:54.385100 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:13:54.385858 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:13:54.385860 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:13:54.386693 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:13:54.386696 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:13:54.387053 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:13:54.387054 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:13:54.387166 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:54.387716 (+ 550us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 01:13:54.387717 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:13:54.387746 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 01:13:54.387771 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:13:54.388000 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:13:54.388007 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:13:54.388010 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 01:13:54.388088 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 01:13:54.388196 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":71,"threads_started":1}
I20260926 01:13:54.388944 20264 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37517
I20260926 01:13:54.389048 20264 heartbeater.cc:464] Registering TS with master...
I20260926 01:13:54.389237 20264 heartbeater.cc:511] Master 127.0.98.62:37517 requested a full tablet report, sending...
I20260926 01:13:54.389757 20076 ts_manager.cc:194] Registered new tserver with Master: 5f581ddfbed04dd3b39112156731886d (127.0.98.1:37089)
I20260926 01:13:54.390478 20076 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:35727
I20260926 01:13:54.391206 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 20136
I20260926 01:13:54.391288 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384993574762-392-0/minicluster-data/ts-0/wal/instance
I20260926 01:13:54.391469 392 external_mini_cluster.cc:1428] Setting key 03e784c9081177ee9c3681d65132c6e9
I20260926 01:13:54.392387 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384993574762-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384993574762-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384993574762-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384993574762-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37517
--builtin_ntp_servers=127.0.98.20:44431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 01:13:54.471877 20267 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:13:54.472091 20267 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:13:54.472113 20267 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:13:54.473294 20267 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:13:54.473340 20267 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:13:54.473366 20267 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 01:13:54.474763 20267 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384993574762-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384993574762-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384993574762-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37517
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20267
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384993574762-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:13:54.474989 20267 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:13:54.475287 20267 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:13:54.480157 20272 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:13:54.480255 20267 server_base.cc:1034] running on GCE node
W20260926 01:13:54.480195 20273 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:13:54.480165 20275 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:13:54.480600 20267 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:13:54.480832 20267 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:13:54.482025 20267 hybrid_clock.cc:648] HybridClock initialized: now 1790385234482011 us; error 29 us; skew 500 ppm
I20260926 01:13:54.483392 20267 webserver.cc:460] Webserver started at http://127.0.98.2:37643/ using document root <none> and password file <none>
I20260926 01:13:54.483611 20267 fs_manager.cc:359] Metadata directory not provided
I20260926 01:13:54.483659 20267 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:13:54.483781 20267 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:13:54.484829 20267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384993574762-392-0/minicluster-data/ts-1/data/instance:
uuid: "cfad56518bef45ddb37d71c9755a0342"
format_stamp: "Formatted at 2026-09-26 01:13:54 on dist-test-slave-7b22"
server_key: "bf4450e89c4104078a8b0164934e8b55"
server_key_iv: "e187da84d65313d167fea921dd165a4c"
server_key_version: "encryptionkey@0"
I20260926 01:13:54.485159 20267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384993574762-392-0/minicluster-data/ts-1/wal/instance:
uuid: "cfad56518bef45ddb37d71c9755a0342"
format_stamp: "Formatted at 2026-09-26 01:13:54 on dist-test-slave-7b22"
server_key: "bf4450e89c4104078a8b0164934e8b55"
server_key_iv: "e187da84d65313d167fea921dd165a4c"
server_key_version: "encryptionkey@0"
I20260926 01:13:54.486547 20267 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 01:13:54.487294 20281 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:13:54.487447 20267 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 01:13:54.487524 20267 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384993574762-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384993574762-392-0/minicluster-data/ts-1/wal
uuid: "cfad56518bef45ddb37d71c9755a0342"
format_stamp: "Formatted at 2026-09-26 01:13:54 on dist-test-slave-7b22"
server_key: "bf4450e89c4104078a8b0164934e8b55"
server_key_iv: "e187da84d65313d167fea921dd165a4c"
server_key_version: "encryptionkey@0"
I20260926 01:13:54.487586 20267 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384993574762-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384993574762-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384993574762-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:13:54.498670 20267 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:13:54.499054 20267 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:13:54.499236 20267 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:13:54.499482 20267 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:13:54.499899 20267 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:13:54.499944 20267 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:13:54.499972 20267 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:13:54.499989 20267 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:13:54.506016 20267 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36281
I20260926 01:13:54.506069 20394 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36281 every 8 connection(s)
I20260926 01:13:54.506385 20267 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384993574762-392-0/minicluster-data/ts-1/data/info.pb
I20260926 01:13:54.506999 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 20267
I20260926 01:13:54.507107 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384993574762-392-0/minicluster-data/ts-1/wal/instance
I20260926 01:13:54.507300 392 external_mini_cluster.cc:1428] Setting key 956e7ac2b66b2e2da0a12b4eb964a17f
I20260926 01:13:54.508216 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384993574762-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384993574762-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384993574762-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384993574762-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37517
--builtin_ntp_servers=127.0.98.20:44431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 01:13:54.512156 20120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:54.507456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37769 (local address 127.0.98.62:37517)
0926 01:13:54.507502 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 01:13:54.507504 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:13:54.507515 (+ 11us) server_negotiation.cc:407] Connection header received
0926 01:13:54.508077 (+ 562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:13:54.508078 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:13:54.508098 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:13:54.508171 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:13:54.508878 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:54.509571 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:54.510713 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:54.510921 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:54.510975 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:54.511869 (+ 894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:13:54.511870 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:13:54.511871 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:13:54.511875 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:13:54.511891 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:13:54.511895 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:13:54.511940 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0926 01:13:54.512020 (+ 80us) server_negotiation.cc:299] Negotiation successful
0926 01:13:54.512061 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 01:13:54.512348 20397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:54.507173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37517 (local address 127.0.98.2:37769)
0926 01:13:54.507355 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0926 01:13:54.507400 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 01:13:54.507981 (+ 581us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:13:54.508236 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:13:54.508239 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:13:54.508247 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:13:54.508795 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:13:54.508799 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:13:54.509664 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:13:54.509666 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:13:54.510554 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:13:54.510557 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:13:54.510990 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:13:54.510991 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:13:54.511133 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:54.511717 (+ 584us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 01:13:54.511717 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:13:54.511747 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 01:13:54.511775 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:13:54.511947 (+ 172us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:13:54.511953 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:13:54.511956 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 01:13:54.512014 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 01:13:54.512118 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":79,"threads_started":1}
I20260926 01:13:54.512858 20395 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37517
I20260926 01:13:54.512951 20395 heartbeater.cc:464] Registering TS with master...
I20260926 01:13:54.513125 20395 heartbeater.cc:511] Master 127.0.98.62:37517 requested a full tablet report, sending...
I20260926 01:13:54.513505 20076 ts_manager.cc:194] Registered new tserver with Master: cfad56518bef45ddb37d71c9755a0342 (127.0.98.2:36281)
I20260926 01:13:54.514021 20076 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:37769
W20260926 01:13:54.589792 20398 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:13:54.590058 20398 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:13:54.590090 20398 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:13:54.591280 20398 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:13:54.591320 20398 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:13:54.591343 20398 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 01:13:54.592690 20398 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384993574762-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384993574762-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384993574762-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37517
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20398
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384993574762-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:13:54.592880 20398 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:13:54.593129 20398 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 01:13:54.598049 20398 server_base.cc:1034] running on GCE node
W20260926 01:13:54.598006 20406 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:13:54.598001 20403 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:13:54.598001 20404 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:13:54.598456 20398 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:13:54.598711 20398 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:13:54.599869 20398 hybrid_clock.cc:648] HybridClock initialized: now 1790385234599850 us; error 28 us; skew 500 ppm
I20260926 01:13:54.601153 20398 webserver.cc:460] Webserver started at http://127.0.98.3:39667/ using document root <none> and password file <none>
I20260926 01:13:54.601394 20398 fs_manager.cc:359] Metadata directory not provided
I20260926 01:13:54.601445 20398 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:13:54.601572 20398 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:13:54.602723 20398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384993574762-392-0/minicluster-data/ts-2/data/instance:
uuid: "2152dd440740432c8ba2979a738b2dd5"
format_stamp: "Formatted at 2026-09-26 01:13:54 on dist-test-slave-7b22"
server_key: "f8f3698aaee2f725bfe07e3aa4191b0a"
server_key_iv: "c3d2d2d93397f8f7085b0c2bfdb02099"
server_key_version: "encryptionkey@0"
I20260926 01:13:54.603052 20398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384993574762-392-0/minicluster-data/ts-2/wal/instance:
uuid: "2152dd440740432c8ba2979a738b2dd5"
format_stamp: "Formatted at 2026-09-26 01:13:54 on dist-test-slave-7b22"
server_key: "f8f3698aaee2f725bfe07e3aa4191b0a"
server_key_iv: "c3d2d2d93397f8f7085b0c2bfdb02099"
server_key_version: "encryptionkey@0"
I20260926 01:13:54.604282 20398 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 01:13:54.604979 20412 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:13:54.605140 20398 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 01:13:54.605218 20398 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384993574762-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384993574762-392-0/minicluster-data/ts-2/wal
uuid: "2152dd440740432c8ba2979a738b2dd5"
format_stamp: "Formatted at 2026-09-26 01:13:54 on dist-test-slave-7b22"
server_key: "f8f3698aaee2f725bfe07e3aa4191b0a"
server_key_iv: "c3d2d2d93397f8f7085b0c2bfdb02099"
server_key_version: "encryptionkey@0"
I20260926 01:13:54.605286 20398 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384993574762-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384993574762-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384993574762-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:13:54.621709 20398 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:13:54.622090 20398 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:13:54.622218 20398 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:13:54.622440 20398 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:13:54.622761 20398 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:13:54.622807 20398 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:13:54.622836 20398 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:13:54.622852 20398 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:13:54.628091 20398 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36715
I20260926 01:13:54.628175 20525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36715 every 8 connection(s)
I20260926 01:13:54.628444 20398 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384993574762-392-0/minicluster-data/ts-2/data/info.pb
I20260926 01:13:54.633131 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 20398
I20260926 01:13:54.633262 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384993574762-392-0/minicluster-data/ts-2/wal/instance
I20260926 01:13:54.633437 392 external_mini_cluster.cc:1428] Setting key d2d943a084c8dd0f95ca54108e333120
I20260926 01:13:54.633737 20120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:54.629194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39265 (local address 127.0.98.62:37517)
0926 01:13:54.629284 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0926 01:13:54.629287 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:13:54.629460 (+ 173us) server_negotiation.cc:407] Connection header received
0926 01:13:54.630083 (+ 623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:13:54.630084 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:13:54.630097 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:13:54.630148 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:13:54.630834 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:54.631348 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:54.632311 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:54.632511 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:54.632566 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:54.633405 (+ 839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:13:54.633406 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:13:54.633406 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:13:54.633412 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:13:54.633426 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:13:54.633430 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:13:54.633475 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0926 01:13:54.633571 (+ 96us) server_negotiation.cc:299] Negotiation successful
0926 01:13:54.633649 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 01:13:54.633915 20528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:54.629126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37517 (local address 127.0.98.3:39265)
0926 01:13:54.629342 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0926 01:13:54.629376 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 01:13:54.629974 (+ 598us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:13:54.630220 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:13:54.630224 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:13:54.630234 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:13:54.630745 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:13:54.630749 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:13:54.631431 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:13:54.631432 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:13:54.632226 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:13:54.632228 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:13:54.632582 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:13:54.632583 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:13:54.632722 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:54.633256 (+ 534us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 01:13:54.633257 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:13:54.633297 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 01:13:54.633327 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:13:54.633485 (+ 158us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:13:54.633491 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:13:54.633494 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 01:13:54.633559 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 01:13:54.633678 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":87,"threads_started":1}
I20260926 01:13:54.634408 20526 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37517
I20260926 01:13:54.634502 20526 heartbeater.cc:464] Registering TS with master...
I20260926 01:13:54.634688 20526 heartbeater.cc:511] Master 127.0.98.62:37517 requested a full tablet report, sending...
I20260926 01:13:54.635072 20076 ts_manager.cc:194] Registered new tserver with Master: 2152dd440740432c8ba2979a738b2dd5 (127.0.98.3:36715)
I20260926 01:13:54.635476 20076 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:39265
I20260926 01:13:54.645649 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 01:13:54.650367 20120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:54.647125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60538 (local address 127.0.98.62:37517)
0926 01:13:54.647177 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 01:13:54.647180 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:13:54.647238 (+ 58us) server_negotiation.cc:407] Connection header received
0926 01:13:54.647311 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:13:54.647312 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:13:54.647326 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:13:54.647384 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:13:54.647513 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:54.648317 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:54.649062 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:54.649366 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:54.649437 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:54.650111 (+ 674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:13:54.650113 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:13:54.650113 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:13:54.650118 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:13:54.650136 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:13:54.650140 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:13:54.650196 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 01:13:54.650235 (+ 39us) server_negotiation.cc:299] Negotiation successful
0926 01:13:54.650263 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 01:13:54.651937 20076 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60538:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:13:54.653434 20076 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:13:54.659525 20537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:54.655603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36715 (local address 127.0.0.1:53354)
0926 01:13:54.656234 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0926 01:13:54.656244 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 01:13:54.656312 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:13:54.656537 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:13:54.656538 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:13:54.656539 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:13:54.656601 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:13:54.656602 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:13:54.657736 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:13:54.657737 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:13:54.658635 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:13:54.658637 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:13:54.659287 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:13:54.659288 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:13:54.659346 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:54.659350 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 01:13:54.659400 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 01:13:54.659437 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"thread_start_us":136,"threads_started":1}
I20260926 01:13:54.659524 20538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:54.655695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53354 (local address 127.0.98.3:36715)
0926 01:13:54.656177 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0926 01:13:54.656181 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 01:13:54.656288 (+ 107us) server_negotiation.cc:407] Connection header received
0926 01:13:54.656393 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:13:54.656395 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:13:54.656408 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:13:54.656456 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:13:54.656687 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:54.657664 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:54.658711 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:54.659207 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:54.659276 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:54.659360 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 01:13:54.659413 (+ 53us) server_negotiation.cc:299] Negotiation successful
0926 01:13:54.659449 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":60,"threads_started":1}
I20260926 01:13:54.659819 20536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:54.655509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60188 (local address 127.0.98.1:37089)
0926 01:13:54.655872 (+ 363us) server_negotiation.cc:206] Beginning negotiation
0926 01:13:54.655877 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 01:13:54.655888 (+ 11us) server_negotiation.cc:407] Connection header received
0926 01:13:54.655932 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:13:54.655934 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:13:54.655949 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:13:54.656038 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:13:54.656200 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:54.657146 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:54.658585 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:54.659117 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:54.659174 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:54.659249 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 01:13:54.659486 (+ 237us) server_negotiation.cc:299] Negotiation successful
0926 01:13:54.659529 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":98,"threads_started":1}
I20260926 01:13:54.659974 20534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:54.655412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37089 (local address 127.0.0.1:60188)
0926 01:13:54.655597 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 01:13:54.655637 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 01:13:54.655738 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:13:54.656045 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:13:54.656049 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:13:54.656051 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:13:54.656129 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:13:54.656130 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:13:54.657216 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:13:54.657217 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:13:54.658494 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:13:54.658496 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:13:54.659197 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:13:54.659198 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:13:54.659274 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:54.659288 (+ 14us) client_negotiation.cc:769] Sending connection context
0926 01:13:54.659350 (+ 62us) client_negotiation.cc:240] Negotiation successful
0926 01:13:54.659402 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":88,"threads_started":1}
I20260926 01:13:54.660343 20535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:54.655415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36281 (local address 127.0.0.1:55234)
0926 01:13:54.655903 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0926 01:13:54.655911 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 01:13:54.655954 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:13:54.656084 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:13:54.656085 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:13:54.656086 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:13:54.656232 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:13:54.656234 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:13:54.657439 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:13:54.657440 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:13:54.658500 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:13:54.658501 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:13:54.660106 (+ 1605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:13:54.660108 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:13:54.660180 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:54.660184 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 01:13:54.660242 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 01:13:54.660283 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"mutex_wait_us":5,"thread_start_us":59,"threads_started":1}
I20260926 01:13:54.660264 20460 tablet_service.cc:1467] Processing CreateTablet for tablet b30babb549cd4837bd80dcc5d1b31adb (DEFAULT_TABLE table=test-table [id=beb0fe56967b45859922cdf278f2b79c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:13:54.660548 20539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:54.655721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55234 (local address 127.0.98.2:36281)
0926 01:13:54.655894 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 01:13:54.655898 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 01:13:54.655962 (+ 64us) server_negotiation.cc:407] Connection header received
0926 01:13:54.656000 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:13:54.656002 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:13:54.656015 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:13:54.656083 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:13:54.656306 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:54.657360 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:54.659285 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:54.659964 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:54.660100 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:54.660189 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 01:13:54.660253 (+ 64us) server_negotiation.cc:299] Negotiation successful
0926 01:13:54.660300 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":87,"threads_started":1}
I20260926 01:13:54.660629 20460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b30babb549cd4837bd80dcc5d1b31adb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:13:54.661113 20329 tablet_service.cc:1467] Processing CreateTablet for tablet b30babb549cd4837bd80dcc5d1b31adb (DEFAULT_TABLE table=test-table [id=beb0fe56967b45859922cdf278f2b79c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:13:54.661391 20329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b30babb549cd4837bd80dcc5d1b31adb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:13:54.662458 20198 tablet_service.cc:1467] Processing CreateTablet for tablet b30babb549cd4837bd80dcc5d1b31adb (DEFAULT_TABLE table=test-table [id=beb0fe56967b45859922cdf278f2b79c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:13:54.662784 20198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b30babb549cd4837bd80dcc5d1b31adb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:13:54.663327 20540 tablet_bootstrap.cc:492] T b30babb549cd4837bd80dcc5d1b31adb P 2152dd440740432c8ba2979a738b2dd5: Bootstrap starting.
I20260926 01:13:54.663938 20540 tablet_bootstrap.cc:654] T b30babb549cd4837bd80dcc5d1b31adb P 2152dd440740432c8ba2979a738b2dd5: Neither blocks nor log segments found. Creating new log.
I20260926 01:13:54.664206 20540 log.cc:824] T b30babb549cd4837bd80dcc5d1b31adb P 2152dd440740432c8ba2979a738b2dd5: Log is configured to *not* fsync() on all Append() calls
I20260926 01:13:54.664420 20541 tablet_bootstrap.cc:492] T b30babb549cd4837bd80dcc5d1b31adb P cfad56518bef45ddb37d71c9755a0342: Bootstrap starting.
I20260926 01:13:54.664952 20541 tablet_bootstrap.cc:654] T b30babb549cd4837bd80dcc5d1b31adb P cfad56518bef45ddb37d71c9755a0342: Neither blocks nor log segments found. Creating new log.
I20260926 01:13:54.665199 20541 log.cc:824] T b30babb549cd4837bd80dcc5d1b31adb P cfad56518bef45ddb37d71c9755a0342: Log is configured to *not* fsync() on all Append() calls
I20260926 01:13:54.665648 20544 tablet_bootstrap.cc:492] T b30babb549cd4837bd80dcc5d1b31adb P 5f581ddfbed04dd3b39112156731886d: Bootstrap starting.
I20260926 01:13:54.665807 20541 tablet_bootstrap.cc:492] T b30babb549cd4837bd80dcc5d1b31adb P cfad56518bef45ddb37d71c9755a0342: No bootstrap required, opened a new log
I20260926 01:13:54.665896 20541 ts_tablet_manager.cc:1397] T b30babb549cd4837bd80dcc5d1b31adb P cfad56518bef45ddb37d71c9755a0342: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 01:13:54.666162 20540 tablet_bootstrap.cc:492] T b30babb549cd4837bd80dcc5d1b31adb P 2152dd440740432c8ba2979a738b2dd5: No bootstrap required, opened a new log
I20260926 01:13:54.666178 20544 tablet_bootstrap.cc:654] T b30babb549cd4837bd80dcc5d1b31adb P 5f581ddfbed04dd3b39112156731886d: Neither blocks nor log segments found. Creating new log.
I20260926 01:13:54.666213 20540 ts_tablet_manager.cc:1397] T b30babb549cd4837bd80dcc5d1b31adb P 2152dd440740432c8ba2979a738b2dd5: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260926 01:13:54.666383 20544 log.cc:824] T b30babb549cd4837bd80dcc5d1b31adb P 5f581ddfbed04dd3b39112156731886d: Log is configured to *not* fsync() on all Append() calls
I20260926 01:13:54.666913 20544 tablet_bootstrap.cc:492] T b30babb549cd4837bd80dcc5d1b31adb P 5f581ddfbed04dd3b39112156731886d: No bootstrap required, opened a new log
I20260926 01:13:54.666952 20544 ts_tablet_manager.cc:1397] T b30babb549cd4837bd80dcc5d1b31adb P 5f581ddfbed04dd3b39112156731886d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 01:13:54.668337 20541 raft_consensus.cc:348] T b30babb549cd4837bd80dcc5d1b31adb P cfad56518bef45ddb37d71c9755a0342 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfad56518bef45ddb37d71c9755a0342" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36281 } } peers { permanent_uuid: "5f581ddfbed04dd3b39112156731886d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37089 } } peers { permanent_uuid: "2152dd440740432c8ba2979a738b2dd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36715 } }
I20260926 01:13:54.668496 20541 raft_consensus.cc:374] T b30babb549cd4837bd80dcc5d1b31adb P cfad56518bef45ddb37d71c9755a0342 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:13:54.668526 20541 raft_consensus.cc:729] T b30babb549cd4837bd80dcc5d1b31adb P cfad56518bef45ddb37d71c9755a0342 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfad56518bef45ddb37d71c9755a0342, State: Initialized, Role: FOLLOWER
I20260926 01:13:54.668655 20541 consensus_queue.cc:260] T b30babb549cd4837bd80dcc5d1b31adb P cfad56518bef45ddb37d71c9755a0342 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfad56518bef45ddb37d71c9755a0342" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36281 } } peers { permanent_uuid: "5f581ddfbed04dd3b39112156731886d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37089 } } peers { permanent_uuid: "2152dd440740432c8ba2979a738b2dd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36715 } }
I20260926 01:13:54.668700 20540 raft_consensus.cc:348] T b30babb549cd4837bd80dcc5d1b31adb P 2152dd440740432c8ba2979a738b2dd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfad56518bef45ddb37d71c9755a0342" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36281 } } peers { permanent_uuid: "5f581ddfbed04dd3b39112156731886d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37089 } } peers { permanent_uuid: "2152dd440740432c8ba2979a738b2dd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36715 } }
I20260926 01:13:54.668829 20540 raft_consensus.cc:374] T b30babb549cd4837bd80dcc5d1b31adb P 2152dd440740432c8ba2979a738b2dd5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:13:54.668859 20540 raft_consensus.cc:729] T b30babb549cd4837bd80dcc5d1b31adb P 2152dd440740432c8ba2979a738b2dd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2152dd440740432c8ba2979a738b2dd5, State: Initialized, Role: FOLLOWER
I20260926 01:13:54.668891 20541 ts_tablet_manager.cc:1428] T b30babb549cd4837bd80dcc5d1b31adb P cfad56518bef45ddb37d71c9755a0342: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260926 01:13:54.668957 20540 consensus_queue.cc:260] T b30babb549cd4837bd80dcc5d1b31adb P 2152dd440740432c8ba2979a738b2dd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfad56518bef45ddb37d71c9755a0342" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36281 } } peers { permanent_uuid: "5f581ddfbed04dd3b39112156731886d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37089 } } peers { permanent_uuid: "2152dd440740432c8ba2979a738b2dd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36715 } }
I20260926 01:13:54.669021 20395 heartbeater.cc:503] Master 127.0.98.62:37517 was elected leader, sending a full tablet report...
I20260926 01:13:54.669049 20544 raft_consensus.cc:348] T b30babb549cd4837bd80dcc5d1b31adb P 5f581ddfbed04dd3b39112156731886d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfad56518bef45ddb37d71c9755a0342" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36281 } } peers { permanent_uuid: "5f581ddfbed04dd3b39112156731886d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37089 } } peers { permanent_uuid: "2152dd440740432c8ba2979a738b2dd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36715 } }
I20260926 01:13:54.669173 20540 ts_tablet_manager.cc:1428] T b30babb549cd4837bd80dcc5d1b31adb P 2152dd440740432c8ba2979a738b2dd5: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 01:13:54.669188 20544 raft_consensus.cc:374] T b30babb549cd4837bd80dcc5d1b31adb P 5f581ddfbed04dd3b39112156731886d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:13:54.669224 20544 raft_consensus.cc:729] T b30babb549cd4837bd80dcc5d1b31adb P 5f581ddfbed04dd3b39112156731886d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f581ddfbed04dd3b39112156731886d, State: Initialized, Role: FOLLOWER
I20260926 01:13:54.669319 20544 consensus_queue.cc:260] T b30babb549cd4837bd80dcc5d1b31adb P 5f581ddfbed04dd3b39112156731886d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfad56518bef45ddb37d71c9755a0342" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36281 } } peers { permanent_uuid: "5f581ddfbed04dd3b39112156731886d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37089 } } peers { permanent_uuid: "2152dd440740432c8ba2979a738b2dd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36715 } }
I20260926 01:13:54.669507 20526 heartbeater.cc:503] Master 127.0.98.62:37517 was elected leader, sending a full tablet report...
I20260926 01:13:54.669550 20544 ts_tablet_manager.cc:1428] T b30babb549cd4837bd80dcc5d1b31adb P 5f581ddfbed04dd3b39112156731886d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 01:13:54.669778 20264 heartbeater.cc:503] Master 127.0.98.62:37517 was elected leader, sending a full tablet report...
W20260926 01:13:54.756976 20396 tablet.cc:2369] T b30babb549cd4837bd80dcc5d1b31adb P cfad56518bef45ddb37d71c9755a0342: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 01:13:54.879017 20527 tablet.cc:2369] T b30babb549cd4837bd80dcc5d1b31adb P 2152dd440740432c8ba2979a738b2dd5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 01:13:54.883445 20265 tablet.cc:2369] T b30babb549cd4837bd80dcc5d1b31adb P 5f581ddfbed04dd3b39112156731886d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 01:13:54.913089 20547 raft_consensus.cc:482] T b30babb549cd4837bd80dcc5d1b31adb P 2152dd440740432c8ba2979a738b2dd5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:13:54.913231 20547 raft_consensus.cc:504] T b30babb549cd4837bd80dcc5d1b31adb P 2152dd440740432c8ba2979a738b2dd5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfad56518bef45ddb37d71c9755a0342" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36281 } } peers { permanent_uuid: "5f581ddfbed04dd3b39112156731886d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37089 } } peers { permanent_uuid: "2152dd440740432c8ba2979a738b2dd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36715 } }
I20260926 01:13:54.913559 20547 leader_election.cc:288] T b30babb549cd4837bd80dcc5d1b31adb P 2152dd440740432c8ba2979a738b2dd5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cfad56518bef45ddb37d71c9755a0342 (127.0.98.2:36281), 5f581ddfbed04dd3b39112156731886d (127.0.98.1:37089)
I20260926 01:13:54.917640 20536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:54.913903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39297 (local address 127.0.98.1:37089)
0926 01:13:54.913981 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0926 01:13:54.913982 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 01:13:54.914051 (+ 69us) server_negotiation.cc:407] Connection header received
0926 01:13:54.914095 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:13:54.914097 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:13:54.914108 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:13:54.914166 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:13:54.914303 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:54.915023 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:54.916584 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:54.917183 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:54.917276 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:54.917285 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 01:13:54.917521 (+ 236us) server_negotiation.cc:299] Negotiation successful
0926 01:13:54.917567 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 01:13:54.917672 20528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:54.913760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36281 (local address 127.0.98.3:47033)
0926 01:13:54.913843 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0926 01:13:54.913854 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 01:13:54.913899 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:13:54.914074 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:13:54.914075 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:13:54.914076 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:13:54.914156 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:13:54.914159 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:13:54.915157 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:13:54.915158 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:13:54.916453 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:13:54.916457 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:13:54.917290 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:13:54.917292 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:13:54.917384 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:54.917390 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 01:13:54.917454 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 01:13:54.917509 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":37}
I20260926 01:13:54.917646 20539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:54.913871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47033 (local address 127.0.98.2:36281)
0926 01:13:54.913922 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 01:13:54.913924 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:13:54.913935 (+ 11us) server_negotiation.cc:407] Connection header received
0926 01:13:54.913973 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:13:54.913974 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:13:54.913992 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:13:54.914057 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:13:54.914228 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:54.915044 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:54.916617 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:54.917185 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:54.917265 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:54.917277 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0926 01:13:54.917512 (+ 235us) server_negotiation.cc:299] Negotiation successful
0926 01:13:54.917564 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 01:13:54.917678 20549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:54.913788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37089 (local address 127.0.98.3:39297)
0926 01:13:54.913988 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0926 01:13:54.913995 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0926 01:13:54.914043 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:13:54.914170 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:13:54.914171 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:13:54.914172 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:13:54.914239 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:13:54.914241 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:13:54.915155 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:13:54.915156 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:13:54.916460 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:13:54.916462 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:13:54.917308 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:13:54.917308 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:13:54.917404 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:54.917407 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 01:13:54.917477 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 01:13:54.917515 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"mutex_wait_us":43,"thread_start_us":118,"threads_started":1}
I20260926 01:13:54.918138 20218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b30babb549cd4837bd80dcc5d1b31adb" candidate_uuid: "2152dd440740432c8ba2979a738b2dd5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f581ddfbed04dd3b39112156731886d" is_pre_election: true
I20260926 01:13:54.918138 20349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b30babb549cd4837bd80dcc5d1b31adb" candidate_uuid: "2152dd440740432c8ba2979a738b2dd5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfad56518bef45ddb37d71c9755a0342" is_pre_election: true
I20260926 01:13:54.918305 20218 raft_consensus.cc:2454] T b30babb549cd4837bd80dcc5d1b31adb P 5f581ddfbed04dd3b39112156731886d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2152dd440740432c8ba2979a738b2dd5 in term 0.
I20260926 01:13:54.918330 20349 raft_consensus.cc:2454] T b30babb549cd4837bd80dcc5d1b31adb P cfad56518bef45ddb37d71c9755a0342 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2152dd440740432c8ba2979a738b2dd5 in term 0.
I20260926 01:13:54.918547 20416 leader_election.cc:302] T b30babb549cd4837bd80dcc5d1b31adb P 2152dd440740432c8ba2979a738b2dd5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2152dd440740432c8ba2979a738b2dd5, 5f581ddfbed04dd3b39112156731886d; no voters:
I20260926 01:13:54.918762 20547 raft_consensus.cc:2781] T b30babb549cd4837bd80dcc5d1b31adb P 2152dd440740432c8ba2979a738b2dd5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:13:54.918825 20547 raft_consensus.cc:482] T b30babb549cd4837bd80dcc5d1b31adb P 2152dd440740432c8ba2979a738b2dd5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:13:54.918843 20547 raft_consensus.cc:3037] T b30babb549cd4837bd80dcc5d1b31adb P 2152dd440740432c8ba2979a738b2dd5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:13:54.919505 20547 raft_consensus.cc:504] T b30babb549cd4837bd80dcc5d1b31adb P 2152dd440740432c8ba2979a738b2dd5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfad56518bef45ddb37d71c9755a0342" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36281 } } peers { permanent_uuid: "5f581ddfbed04dd3b39112156731886d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37089 } } peers { permanent_uuid: "2152dd440740432c8ba2979a738b2dd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36715 } }
I20260926 01:13:54.919632 20547 leader_election.cc:288] T b30babb549cd4837bd80dcc5d1b31adb P 2152dd440740432c8ba2979a738b2dd5 [CANDIDATE]: Term 1 election: Requested vote from peers cfad56518bef45ddb37d71c9755a0342 (127.0.98.2:36281), 5f581ddfbed04dd3b39112156731886d (127.0.98.1:37089)
I20260926 01:13:54.919835 20218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b30babb549cd4837bd80dcc5d1b31adb" candidate_uuid: "2152dd440740432c8ba2979a738b2dd5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f581ddfbed04dd3b39112156731886d"
I20260926 01:13:54.919842 20349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b30babb549cd4837bd80dcc5d1b31adb" candidate_uuid: "2152dd440740432c8ba2979a738b2dd5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfad56518bef45ddb37d71c9755a0342"
I20260926 01:13:54.919911 20218 raft_consensus.cc:3037] T b30babb549cd4837bd80dcc5d1b31adb P 5f581ddfbed04dd3b39112156731886d [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:13:54.919930 20349 raft_consensus.cc:3037] T b30babb549cd4837bd80dcc5d1b31adb P cfad56518bef45ddb37d71c9755a0342 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:13:54.920603 20218 raft_consensus.cc:2454] T b30babb549cd4837bd80dcc5d1b31adb P 5f581ddfbed04dd3b39112156731886d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2152dd440740432c8ba2979a738b2dd5 in term 1.
I20260926 01:13:54.920696 20349 raft_consensus.cc:2454] T b30babb549cd4837bd80dcc5d1b31adb P cfad56518bef45ddb37d71c9755a0342 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2152dd440740432c8ba2979a738b2dd5 in term 1.
I20260926 01:13:54.920778 20416 leader_election.cc:302] T b30babb549cd4837bd80dcc5d1b31adb P 2152dd440740432c8ba2979a738b2dd5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2152dd440740432c8ba2979a738b2dd5, 5f581ddfbed04dd3b39112156731886d; no voters:
I20260926 01:13:54.920868 20547 raft_consensus.cc:2781] T b30babb549cd4837bd80dcc5d1b31adb P 2152dd440740432c8ba2979a738b2dd5 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:13:54.921025 20547 raft_consensus.cc:686] T b30babb549cd4837bd80dcc5d1b31adb P 2152dd440740432c8ba2979a738b2dd5 [term 1 LEADER]: Becoming Leader. State: Replica: 2152dd440740432c8ba2979a738b2dd5, State: Running, Role: LEADER
I20260926 01:13:54.921133 20547 consensus_queue.cc:237] T b30babb549cd4837bd80dcc5d1b31adb P 2152dd440740432c8ba2979a738b2dd5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfad56518bef45ddb37d71c9755a0342" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36281 } } peers { permanent_uuid: "5f581ddfbed04dd3b39112156731886d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37089 } } peers { permanent_uuid: "2152dd440740432c8ba2979a738b2dd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36715 } }
I20260926 01:13:54.922046 20075 catalog_manager.cc:5495] T b30babb549cd4837bd80dcc5d1b31adb P 2152dd440740432c8ba2979a738b2dd5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2152dd440740432c8ba2979a738b2dd5 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "2152dd440740432c8ba2979a738b2dd5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfad56518bef45ddb37d71c9755a0342" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36281 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f581ddfbed04dd3b39112156731886d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2152dd440740432c8ba2979a738b2dd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36715 } health_report { overall_health: HEALTHY } } }
I20260926 01:13:54.945596 20538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:54.943268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53370 (local address 127.0.98.3:36715)
0926 01:13:54.943384 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0926 01:13:54.943386 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:13:54.943396 (+ 10us) server_negotiation.cc:407] Connection header received
0926 01:13:54.943432 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:13:54.943433 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:13:54.943451 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:13:54.943518 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:13:54.943661 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:54.944126 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:54.944930 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:54.945135 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:54.945189 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:54.945323 (+ 134us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:13:54.945424 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:13:54.945482 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 01:13:54.945494 (+ 12us) server_negotiation.cc:299] Negotiation successful
0926 01:13:54.945510 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260926 01:13:54.947660 20218 raft_consensus.cc:1261] T b30babb549cd4837bd80dcc5d1b31adb P 5f581ddfbed04dd3b39112156731886d [term 1 FOLLOWER]: Refusing update from remote peer 2152dd440740432c8ba2979a738b2dd5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:13:54.947659 20349 raft_consensus.cc:1261] T b30babb549cd4837bd80dcc5d1b31adb P cfad56518bef45ddb37d71c9755a0342 [term 1 FOLLOWER]: Refusing update from remote peer 2152dd440740432c8ba2979a738b2dd5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:13:54.948081 20550 consensus_queue.cc:1035] T b30babb549cd4837bd80dcc5d1b31adb P 2152dd440740432c8ba2979a738b2dd5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f581ddfbed04dd3b39112156731886d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:13:54.948223 20547 consensus_queue.cc:1035] T b30babb549cd4837bd80dcc5d1b31adb P 2152dd440740432c8ba2979a738b2dd5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfad56518bef45ddb37d71c9755a0342" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:13:54.951009 20554 mvcc.cc:205] Tried to move back new op lower bound from 7333417922342801408 to 7333417922237538304. Current Snapshot: MvccSnapshot[applied={T|T < 7333417922342801408}]
I20260926 01:13:54.951282 20555 mvcc.cc:205] Tried to move back new op lower bound from 7333417922342801408 to 7333417922237538304. Current Snapshot: MvccSnapshot[applied={T|T < 7333417922342801408}]
I20260926 01:13:54.952841 20076 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60538:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:13:54.952925 20076 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60538:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:13:54.953615 20076 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4ae3adedbc40426f86292abf1ac76001: Sending DeleteTablet for 3 replicas of tablet b30babb549cd4837bd80dcc5d1b31adb
I20260926 01:13:54.953943 20198 tablet_service.cc:1514] Processing DeleteTablet for tablet b30babb549cd4837bd80dcc5d1b31adb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:13:54 UTC) from {username='slave'} at 127.0.0.1:60188
I20260926 01:13:54.953969 20329 tablet_service.cc:1514] Processing DeleteTablet for tablet b30babb549cd4837bd80dcc5d1b31adb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:13:54 UTC) from {username='slave'} at 127.0.0.1:55234
I20260926 01:13:54.954092 20460 tablet_service.cc:1514] Processing DeleteTablet for tablet b30babb549cd4837bd80dcc5d1b31adb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:13:54 UTC) from {username='slave'} at 127.0.0.1:53354
I20260926 01:13:54.954209 20563 tablet_replica.cc:330] T b30babb549cd4837bd80dcc5d1b31adb P 5f581ddfbed04dd3b39112156731886d: stopping tablet replica
I20260926 01:13:54.954313 20563 raft_consensus.cc:2229] T b30babb549cd4837bd80dcc5d1b31adb P 5f581ddfbed04dd3b39112156731886d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:13:54.954339 20565 tablet_replica.cc:330] T b30babb549cd4837bd80dcc5d1b31adb P 2152dd440740432c8ba2979a738b2dd5: stopping tablet replica
I20260926 01:13:54.954432 20563 raft_consensus.cc:2258] T b30babb549cd4837bd80dcc5d1b31adb P 5f581ddfbed04dd3b39112156731886d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:13:54.954540 20565 raft_consensus.cc:2229] T b30babb549cd4837bd80dcc5d1b31adb P 2152dd440740432c8ba2979a738b2dd5 [term 1 LEADER]: Raft consensus shutting down.
I20260926 01:13:54.954586 20564 tablet_replica.cc:330] T b30babb549cd4837bd80dcc5d1b31adb P cfad56518bef45ddb37d71c9755a0342: stopping tablet replica
I20260926 01:13:54.954617 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 20136
I20260926 01:13:54.954689 20564 raft_consensus.cc:2229] T b30babb549cd4837bd80dcc5d1b31adb P cfad56518bef45ddb37d71c9755a0342 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:13:54.954767 20565 raft_consensus.cc:2258] T b30babb549cd4837bd80dcc5d1b31adb P 2152dd440740432c8ba2979a738b2dd5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:13:54.955303 20564 raft_consensus.cc:2258] T b30babb549cd4837bd80dcc5d1b31adb P cfad56518bef45ddb37d71c9755a0342 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:13:54.955504 20565 ts_tablet_manager.cc:1905] T b30babb549cd4837bd80dcc5d1b31adb P 2152dd440740432c8ba2979a738b2dd5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:13:54.955760 20564 ts_tablet_manager.cc:1905] T b30babb549cd4837bd80dcc5d1b31adb P cfad56518bef45ddb37d71c9755a0342: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:13:54.957441 20565 ts_tablet_manager.cc:1918] T b30babb549cd4837bd80dcc5d1b31adb P 2152dd440740432c8ba2979a738b2dd5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:13:54.957459 20564 ts_tablet_manager.cc:1918] T b30babb549cd4837bd80dcc5d1b31adb P cfad56518bef45ddb37d71c9755a0342: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:13:54.957523 20564 log.cc:1196] T b30babb549cd4837bd80dcc5d1b31adb P cfad56518bef45ddb37d71c9755a0342: Deleting WAL directory at /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384993574762-392-0/minicluster-data/ts-1/wal/wals/b30babb549cd4837bd80dcc5d1b31adb
I20260926 01:13:54.957634 20565 log.cc:1196] T b30babb549cd4837bd80dcc5d1b31adb P 2152dd440740432c8ba2979a738b2dd5: Deleting WAL directory at /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384993574762-392-0/minicluster-data/ts-2/wal/wals/b30babb549cd4837bd80dcc5d1b31adb
I20260926 01:13:54.957767 20564 ts_tablet_manager.cc:1939] T b30babb549cd4837bd80dcc5d1b31adb P cfad56518bef45ddb37d71c9755a0342: Deleting consensus metadata
I20260926 01:13:54.958141 20565 ts_tablet_manager.cc:1939] T b30babb549cd4837bd80dcc5d1b31adb P 2152dd440740432c8ba2979a738b2dd5: Deleting consensus metadata
I20260926 01:13:54.958225 20061 catalog_manager.cc:4841] TS cfad56518bef45ddb37d71c9755a0342 (127.0.98.2:36281): tablet b30babb549cd4837bd80dcc5d1b31adb (table test-table [id=beb0fe56967b45859922cdf278f2b79c]) successfully deleted
I20260926 01:13:54.958778 20061 catalog_manager.cc:4841] TS 2152dd440740432c8ba2979a738b2dd5 (127.0.98.3:36715): tablet b30babb549cd4837bd80dcc5d1b31adb (table test-table [id=beb0fe56967b45859922cdf278f2b79c]) successfully deleted
W20260926 01:13:54.959895 20063 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37089 (error 108)
W20260926 01:13:54.961728 20535 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:13:54.960276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37089 (local address 127.0.0.1:60202)
0926 01:13:54.960337 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0926 01:13:54.960349 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 01:13:54.960394 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:13:54.961671 (+ 1277us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37089: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":39}
W20260926 01:13:54.961946 20063 catalog_manager.cc:4584] TS 5f581ddfbed04dd3b39112156731886d (127.0.98.1:37089): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b30babb549cd4837bd80dcc5d1b31adb: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37089: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 01:13:54.962003 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 20267
I20260926 01:13:54.966604 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 20398
I20260926 01:13:54.970692 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 20045
2026-09-26T01:13:54Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (886 ms)
[----------] 6 tests from EncryptionPolicyTest (8353 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790384993574762-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 01:13:55 dist-test-slave-7b22 krb5kdc[20569](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 01:13:55 dist-test-slave-7b22 krb5kdc[20569](info): set up 2 sockets
Sep 26 01:13:55 dist-test-slave-7b22 krb5kdc[20569](info): commencing operation
krb5kdc: starting...
W20260926 01:13:55.921855 392 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.003s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790384993574762-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790384993574762-392-0/krb5kdc/test-user.keytab.
Sep 26 01:13:56 dist-test-slave-7b22 krb5kdc[20569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385236, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T01:13:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:13:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790384993574762-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790384993574762-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 01:13:56.041158 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790384993574762-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790384993574762-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790384993574762-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36359
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790384993574762-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36359
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:13:56.120186 20582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:13:56.120409 20582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:13:56.120434 20582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:13:56.121649 20582 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:13:56.121702 20582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:13:56.121716 20582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:13:56.121726 20582 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:13:56.121737 20582 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:13:56.123298 20582 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790384993574762-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790384993574762-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36359
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790384993574762-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36359
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20582
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790384993574762-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:13:56.123543 20582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:13:56.123767 20582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 01:13:56.128669 20582 server_base.cc:1034] running on GCE node
W20260926 01:13:56.128592 20587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:13:56.128587 20588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:13:56.128592 20590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:13:56.129032 20582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:13:56.129312 20582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:13:56.130450 20582 hybrid_clock.cc:648] HybridClock initialized: now 1790385236130438 us; error 31 us; skew 500 ppm
Sep 26 01:13:56 dist-test-slave-7b22 krb5kdc[20569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385236, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:13:56.132889 20582 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 01:13:56.133333 20582 webserver.cc:460] Webserver started at http://127.0.0.1:41133/ using document root <none> and password file <none>
I20260926 01:13:56.133533 20582 fs_manager.cc:359] Metadata directory not provided
I20260926 01:13:56.133579 20582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:13:56.133695 20582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:13:56.134801 20582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790384993574762-392-0/minicluster-data/master-0/data/instance:
uuid: "4859586532dd4d35b8021585c3d81a5a"
format_stamp: "Formatted at 2026-09-26 01:13:56 on dist-test-slave-7b22"
server_key: "adb0fcbeb06b538e56159248ef447ea9"
server_key_iv: "8d209fa015123780a1a5906f87c637d6"
server_key_version: "encryptionkey@0"
I20260926 01:13:56.135128 20582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790384993574762-392-0/minicluster-data/master-0/wal/instance:
uuid: "4859586532dd4d35b8021585c3d81a5a"
format_stamp: "Formatted at 2026-09-26 01:13:56 on dist-test-slave-7b22"
server_key: "adb0fcbeb06b538e56159248ef447ea9"
server_key_iv: "8d209fa015123780a1a5906f87c637d6"
server_key_version: "encryptionkey@0"
I20260926 01:13:56.136416 20582 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 01:13:56.137226 20597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:13:56.137409 20582 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:13:56.137497 20582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790384993574762-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790384993574762-392-0/minicluster-data/master-0/wal
uuid: "4859586532dd4d35b8021585c3d81a5a"
format_stamp: "Formatted at 2026-09-26 01:13:56 on dist-test-slave-7b22"
server_key: "adb0fcbeb06b538e56159248ef447ea9"
server_key_iv: "8d209fa015123780a1a5906f87c637d6"
server_key_version: "encryptionkey@0"
I20260926 01:13:56.137576 20582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790384993574762-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790384993574762-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790384993574762-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:13:56.161427 20582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:13:56.163570 20582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:13:56.163704 20582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:13:56.167557 20582 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36359
I20260926 01:13:56.167604 20649 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36359 every 8 connection(s)
I20260926 01:13:56.167984 20582 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
I20260926 01:13:56.168809 20650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:13:56.171119 20650 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4859586532dd4d35b8021585c3d81a5a: Bootstrap starting.
I20260926 01:13:56.171689 20650 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4859586532dd4d35b8021585c3d81a5a: Neither blocks nor log segments found. Creating new log.
I20260926 01:13:56.171906 20650 log.cc:824] T 00000000000000000000000000000000 P 4859586532dd4d35b8021585c3d81a5a: Log is configured to *not* fsync() on all Append() calls
I20260926 01:13:56.172547 20650 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4859586532dd4d35b8021585c3d81a5a: No bootstrap required, opened a new log
I20260926 01:13:56.174115 20650 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4859586532dd4d35b8021585c3d81a5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4859586532dd4d35b8021585c3d81a5a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36359 } }
I20260926 01:13:56.174278 20650 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4859586532dd4d35b8021585c3d81a5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:13:56.174302 20650 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4859586532dd4d35b8021585c3d81a5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4859586532dd4d35b8021585c3d81a5a, State: Initialized, Role: FOLLOWER
I20260926 01:13:56.174405 20650 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4859586532dd4d35b8021585c3d81a5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4859586532dd4d35b8021585c3d81a5a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36359 } }
I20260926 01:13:56.174458 20650 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4859586532dd4d35b8021585c3d81a5a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:13:56.174484 20650 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4859586532dd4d35b8021585c3d81a5a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:13:56.174521 20650 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4859586532dd4d35b8021585c3d81a5a [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:13:56.175043 20650 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4859586532dd4d35b8021585c3d81a5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4859586532dd4d35b8021585c3d81a5a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36359 } }
I20260926 01:13:56.175139 20650 leader_election.cc:302] T 00000000000000000000000000000000 P 4859586532dd4d35b8021585c3d81a5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4859586532dd4d35b8021585c3d81a5a; no voters:
I20260926 01:13:56.175304 20650 leader_election.cc:288] T 00000000000000000000000000000000 P 4859586532dd4d35b8021585c3d81a5a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:13:56.175379 20653 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4859586532dd4d35b8021585c3d81a5a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:13:56.175542 20653 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4859586532dd4d35b8021585c3d81a5a [term 1 LEADER]: Becoming Leader. State: Replica: 4859586532dd4d35b8021585c3d81a5a, State: Running, Role: LEADER
I20260926 01:13:56.175549 20650 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4859586532dd4d35b8021585c3d81a5a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:13:56.175642 20653 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4859586532dd4d35b8021585c3d81a5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4859586532dd4d35b8021585c3d81a5a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36359 } }
I20260926 01:13:56.176007 20654 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4859586532dd4d35b8021585c3d81a5a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4859586532dd4d35b8021585c3d81a5a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4859586532dd4d35b8021585c3d81a5a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36359 } } }
I20260926 01:13:56.176090 20654 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4859586532dd4d35b8021585c3d81a5a [sys.catalog]: This master's current role is: LEADER
I20260926 01:13:56.176134 20655 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4859586532dd4d35b8021585c3d81a5a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4859586532dd4d35b8021585c3d81a5a. Latest consensus state: current_term: 1 leader_uuid: "4859586532dd4d35b8021585c3d81a5a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4859586532dd4d35b8021585c3d81a5a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36359 } } }
I20260926 01:13:56.176203 20655 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4859586532dd4d35b8021585c3d81a5a [sys.catalog]: This master's current role is: LEADER
I20260926 01:13:56.176223 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 20582
I20260926 01:13:56.176311 20658 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:13:56.176349 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790384993574762-392-0/minicluster-data/master-0/wal/instance
I20260926 01:13:56.176518 392 external_mini_cluster.cc:1428] Setting key 879ad6949a4179a47c3fb862c56e5483
I20260926 01:13:56.176795 20658 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:13:56.178514 20658 catalog_manager.cc:1347] Generated new cluster ID: 10911ef6decb4341b647db7e2f3ff2de
I20260926 01:13:56.178563 20658 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 01:13:56 dist-test-slave-7b22 krb5kdc[20569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385236, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 01:13:56.185700 20658 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:13:56.186256 20658 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:13:56.188571 20662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:56.177156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39621 (local address 127.0.0.1:36359)
0926 01:13:56.177323 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 01:13:56.177332 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 01:13:56.177363 (+ 31us) server_negotiation.cc:407] Connection header received
0926 01:13:56.177474 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:13:56.177481 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:13:56.177543 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:13:56.178249 (+ 706us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:13:56.178830 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:56.180269 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:56.181270 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:56.181919 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:56.182035 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:56.184782 (+ 2747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:13:56.184787 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:13:56.184787 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:13:56.184801 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:13:56.186796 (+ 1995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:13:56.187234 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:13:56.187237 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:13:56.187237 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:13:56.187350 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:13:56.187547 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:13:56.187548 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:13:56.187548 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:13:56.187820 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:13:56.187987 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0926 01:13:56.188069 (+ 82us) server_negotiation.cc:299] Negotiation successful
0926 01:13:56.188340 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":80,"threads_started":1}
I20260926 01:13:56.189416 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 01:13:56.191522 20658 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4859586532dd4d35b8021585c3d81a5a: Generated new TSK 0
I20260926 01:13:56.191689 20658 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 01:13:56.200305 20662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:56.190955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37983 (local address 127.0.0.1:36359)
0926 01:13:56.190986 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0926 01:13:56.190988 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:13:56.194035 (+ 3047us) server_negotiation.cc:407] Connection header received
0926 01:13:56.194097 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:13:56.194098 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:13:56.194108 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:13:56.194161 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:13:56.194802 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:56.195332 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:56.196390 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:56.196620 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:56.196694 (+ 74us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:56.198045 (+ 1351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:13:56.198046 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:13:56.198047 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:13:56.198054 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:13:56.199271 (+ 1217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:13:56.199659 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:13:56.199659 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:13:56.199659 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:13:56.199705 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:13:56.199891 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:13:56.199892 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:13:56.199892 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:13:56.199983 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:13:56.200054 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 01:13:56.200183 (+ 129us) server_negotiation.cc:299] Negotiation successful
0926 01:13:56.200227 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260926 01:13:56.202390 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 20582
W20260926 01:13:56.206019 20567 connection.cc:535] client connection to 127.0.0.1:36359 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36359): Connection reset by peer (error 104)
2026-09-26T01:13:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1230 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790384993574762-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 01:13:56 dist-test-slave-7b22 krb5kdc[20681](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 01:13:56 dist-test-slave-7b22 krb5kdc[20681](info): set up 2 sockets
Sep 26 01:13:56 dist-test-slave-7b22 krb5kdc[20681](info): commencing operation
krb5kdc: starting...
W20260926 01:13:57.149622 392 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790384993574762-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790384993574762-392-0/krb5kdc/test-user.keytab.
Sep 26 01:13:57 dist-test-slave-7b22 krb5kdc[20681](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385237, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T01:13:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:13:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790384993574762-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790384993574762-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 01:13:57.269061 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790384993574762-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790384993574762-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790384993574762-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37093
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790384993574762-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37093
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:13:57.348764 20694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:13:57.348973 20694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:13:57.348994 20694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:13:57.350257 20694 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:13:57.350303 20694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:13:57.350317 20694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:13:57.350327 20694 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:13:57.350338 20694 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:13:57.351859 20694 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790384993574762-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790384993574762-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37093
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790384993574762-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37093
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20694
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790384993574762-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:13:57.352087 20694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:13:57.352313 20694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 01:13:57.357301 20694 server_base.cc:1034] running on GCE node
W20260926 01:13:57.357250 20702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:13:57.357259 20699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:13:57.357234 20700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:13:57.357651 20694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:13:57.357955 20694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:13:57.359117 20694 hybrid_clock.cc:648] HybridClock initialized: now 1790385237359105 us; error 32 us; skew 500 ppm
Sep 26 01:13:57 dist-test-slave-7b22 krb5kdc[20681](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385237, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:13:57.367000 20694 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 01:13:57.367472 20694 webserver.cc:460] Webserver started at http://127.0.0.1:38187/ using document root <none> and password file <none>
I20260926 01:13:57.367681 20694 fs_manager.cc:359] Metadata directory not provided
I20260926 01:13:57.367722 20694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:13:57.367812 20694 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:13:57.368860 20694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790384993574762-392-0/minicluster-data/master-0/data/instance:
uuid: "0061fc8fd23d4fde9fc268e40b18771f"
format_stamp: "Formatted at 2026-09-26 01:13:57 on dist-test-slave-7b22"
server_key: "97006430048d122148a1885c058285d9"
server_key_iv: "8d188a883387aa4f7c61a4a141ef21da"
server_key_version: "encryptionkey@0"
I20260926 01:13:57.369171 20694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790384993574762-392-0/minicluster-data/master-0/wal/instance:
uuid: "0061fc8fd23d4fde9fc268e40b18771f"
format_stamp: "Formatted at 2026-09-26 01:13:57 on dist-test-slave-7b22"
server_key: "97006430048d122148a1885c058285d9"
server_key_iv: "8d188a883387aa4f7c61a4a141ef21da"
server_key_version: "encryptionkey@0"
I20260926 01:13:57.370512 20694 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 01:13:57.371263 20709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:13:57.371451 20694 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:13:57.371541 20694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790384993574762-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790384993574762-392-0/minicluster-data/master-0/wal
uuid: "0061fc8fd23d4fde9fc268e40b18771f"
format_stamp: "Formatted at 2026-09-26 01:13:57 on dist-test-slave-7b22"
server_key: "97006430048d122148a1885c058285d9"
server_key_iv: "8d188a883387aa4f7c61a4a141ef21da"
server_key_version: "encryptionkey@0"
I20260926 01:13:57.371604 20694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790384993574762-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790384993574762-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790384993574762-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:13:57.385887 20694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:13:57.387948 20694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:13:57.388083 20694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:13:57.391832 20694 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37093
I20260926 01:13:57.391871 20761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37093 every 8 connection(s)
I20260926 01:13:57.392208 20694 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
I20260926 01:13:57.393021 20762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:13:57.394322 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 20694
I20260926 01:13:57.394447 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790384993574762-392-0/minicluster-data/master-0/wal/instance
I20260926 01:13:57.394637 392 external_mini_cluster.cc:1428] Setting key bd2a4e1a2ea7380b628ba2762fa8aff3
I20260926 01:13:57.395697 20762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0061fc8fd23d4fde9fc268e40b18771f: Bootstrap starting.
I20260926 01:13:57.396323 20762 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0061fc8fd23d4fde9fc268e40b18771f: Neither blocks nor log segments found. Creating new log.
I20260926 01:13:57.396585 20762 log.cc:824] T 00000000000000000000000000000000 P 0061fc8fd23d4fde9fc268e40b18771f: Log is configured to *not* fsync() on all Append() calls
I20260926 01:13:57.397313 20762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0061fc8fd23d4fde9fc268e40b18771f: No bootstrap required, opened a new log
I20260926 01:13:57.399221 20762 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0061fc8fd23d4fde9fc268e40b18771f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0061fc8fd23d4fde9fc268e40b18771f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37093 } }
I20260926 01:13:57.399358 20762 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0061fc8fd23d4fde9fc268e40b18771f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:13:57.399389 20762 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0061fc8fd23d4fde9fc268e40b18771f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0061fc8fd23d4fde9fc268e40b18771f, State: Initialized, Role: FOLLOWER
I20260926 01:13:57.399513 20762 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0061fc8fd23d4fde9fc268e40b18771f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0061fc8fd23d4fde9fc268e40b18771f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37093 } }
I20260926 01:13:57.399572 20762 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0061fc8fd23d4fde9fc268e40b18771f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:13:57.399601 20762 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0061fc8fd23d4fde9fc268e40b18771f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:13:57.399644 20762 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0061fc8fd23d4fde9fc268e40b18771f [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:13:57.400080 20765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:57.395205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48871 (local address 127.0.0.1:37093)
0926 01:13:57.395378 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 01:13:57.395386 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 01:13:57.395411 (+ 25us) server_negotiation.cc:407] Connection header received
0926 01:13:57.395849 (+ 438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:13:57.395855 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:13:57.395903 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:13:57.396018 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:13:57.396295 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:57.397398 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:57.398207 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:57.398601 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:57.398705 (+ 104us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:57.399395 (+ 690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:13:57.399398 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:13:57.399398 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:13:57.399590 (+ 192us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:13:57.399642 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:13:57.399657 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:13:57.399734 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 01:13:57.399846 (+ 112us) server_negotiation.cc:299] Negotiation successful
0926 01:13:57.399869 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":78,"threads_started":1}
I20260926 01:13:57.400271 20762 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0061fc8fd23d4fde9fc268e40b18771f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0061fc8fd23d4fde9fc268e40b18771f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37093 } }
I20260926 01:13:57.400370 20762 leader_election.cc:302] T 00000000000000000000000000000000 P 0061fc8fd23d4fde9fc268e40b18771f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0061fc8fd23d4fde9fc268e40b18771f; no voters:
I20260926 01:13:57.400549 20767 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0061fc8fd23d4fde9fc268e40b18771f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:13:57.400535 20762 leader_election.cc:288] T 00000000000000000000000000000000 P 0061fc8fd23d4fde9fc268e40b18771f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:13:57.400741 20767 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0061fc8fd23d4fde9fc268e40b18771f [term 1 LEADER]: Becoming Leader. State: Replica: 0061fc8fd23d4fde9fc268e40b18771f, State: Running, Role: LEADER
I20260926 01:13:57.400864 20762 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0061fc8fd23d4fde9fc268e40b18771f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:13:57.400851 20767 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0061fc8fd23d4fde9fc268e40b18771f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0061fc8fd23d4fde9fc268e40b18771f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37093 } }
I20260926 01:13:57.400996 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 01:13:57.401193 20767 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0061fc8fd23d4fde9fc268e40b18771f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0061fc8fd23d4fde9fc268e40b18771f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0061fc8fd23d4fde9fc268e40b18771f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37093 } } }
I20260926 01:13:57.401257 20767 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0061fc8fd23d4fde9fc268e40b18771f [sys.catalog]: This master's current role is: LEADER
I20260926 01:13:57.401455 20773 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:13:57.402161 20773 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:13:57.402530 20769 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0061fc8fd23d4fde9fc268e40b18771f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0061fc8fd23d4fde9fc268e40b18771f. Latest consensus state: current_term: 1 leader_uuid: "0061fc8fd23d4fde9fc268e40b18771f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0061fc8fd23d4fde9fc268e40b18771f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37093 } } }
I20260926 01:13:57.402597 20769 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0061fc8fd23d4fde9fc268e40b18771f [sys.catalog]: This master's current role is: LEADER
I20260926 01:13:57.405881 20765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:57.402837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58223 (local address 127.0.0.1:37093)
0926 01:13:57.402874 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 01:13:57.402875 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 01:13:57.402971 (+ 96us) server_negotiation.cc:407] Connection header received
0926 01:13:57.403019 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:13:57.403020 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:13:57.403030 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:13:57.403090 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:13:57.403185 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:57.403907 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:57.404644 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:57.404926 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:57.404989 (+ 63us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:57.405604 (+ 615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:13:57.405606 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:13:57.405606 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:13:57.405611 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:13:57.405626 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:13:57.405630 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:13:57.405692 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 01:13:57.405764 (+ 72us) server_negotiation.cc:299] Negotiation successful
0926 01:13:57.405775 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 01:13:57.411015 20773 catalog_manager.cc:1347] Generated new cluster ID: f41d6cb255de489d8f7b0d89316efb85
I20260926 01:13:57.411082 20773 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 01:13:57.423302 20773 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:13:57.423872 20773 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:13:57.427386 20773 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0061fc8fd23d4fde9fc268e40b18771f: Generated new TSK 0
I20260926 01:13:57.427543 20773 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 01:13:57.433900 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 20694
W20260926 01:13:57.437842 20679 connection.cc:535] client connection to 127.0.0.1:37093 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37093): Connection reset by peer (error 104)
2026-09-26T01:13:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1231 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790384993574762-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 01:13:57 dist-test-slave-7b22 krb5kdc[20793](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 01:13:57 dist-test-slave-7b22 krb5kdc[20793](info): set up 2 sockets
Sep 26 01:13:57 dist-test-slave-7b22 krb5kdc[20793](info): commencing operation
krb5kdc: starting...
W20260926 01:13:58.393637 392 mini_kdc.cc:121] Time spent starting KDC: real 0.942s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790384993574762-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790384993574762-392-0/krb5kdc/test-user.keytab.
Sep 26 01:13:58 dist-test-slave-7b22 krb5kdc[20793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385238, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T01:13:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:13:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790384993574762-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790384993574762-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 01:13:58.510500 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790384993574762-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790384993574762-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790384993574762-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41427
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790384993574762-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41427
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:13:58.590583 20806 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:13:58.590821 20806 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:13:58.590847 20806 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:13:58.592095 20806 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:13:58.592144 20806 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:13:58.592157 20806 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:13:58.592168 20806 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:13:58.592182 20806 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:13:58.593734 20806 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790384993574762-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790384993574762-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41427
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790384993574762-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41427
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20806
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790384993574762-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:13:58.593999 20806 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:13:58.594210 20806 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:13:58.599114 20812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:13:58.599157 20811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:13:58.599195 20814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:13:58.599315 20806 server_base.cc:1034] running on GCE node
I20260926 01:13:58.599548 20806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:13:58.599830 20806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:13:58.601011 20806 hybrid_clock.cc:648] HybridClock initialized: now 1790385238600989 us; error 35 us; skew 500 ppm
Sep 26 01:13:58 dist-test-slave-7b22 krb5kdc[20793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385238, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:13:58.603603 20806 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 01:13:58.604025 20806 webserver.cc:460] Webserver started at http://127.0.0.1:34557/ using document root <none> and password file <none>
I20260926 01:13:58.604220 20806 fs_manager.cc:359] Metadata directory not provided
I20260926 01:13:58.604261 20806 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:13:58.604350 20806 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:13:58.605348 20806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790384993574762-392-0/minicluster-data/master-0/data/instance:
uuid: "bc35806311774c829a4c7795fe9aaf5f"
format_stamp: "Formatted at 2026-09-26 01:13:58 on dist-test-slave-7b22"
server_key: "8639c7d28ed0d0bd8e06e847b58ba0b3"
server_key_iv: "9e9dad6d80d916df98047260bf05cccf"
server_key_version: "encryptionkey@0"
I20260926 01:13:58.605638 20806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790384993574762-392-0/minicluster-data/master-0/wal/instance:
uuid: "bc35806311774c829a4c7795fe9aaf5f"
format_stamp: "Formatted at 2026-09-26 01:13:58 on dist-test-slave-7b22"
server_key: "8639c7d28ed0d0bd8e06e847b58ba0b3"
server_key_iv: "9e9dad6d80d916df98047260bf05cccf"
server_key_version: "encryptionkey@0"
I20260926 01:13:58.606871 20806 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 01:13:58.607528 20821 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:13:58.607673 20806 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 01:13:58.607748 20806 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790384993574762-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790384993574762-392-0/minicluster-data/master-0/wal
uuid: "bc35806311774c829a4c7795fe9aaf5f"
format_stamp: "Formatted at 2026-09-26 01:13:58 on dist-test-slave-7b22"
server_key: "8639c7d28ed0d0bd8e06e847b58ba0b3"
server_key_iv: "9e9dad6d80d916df98047260bf05cccf"
server_key_version: "encryptionkey@0"
I20260926 01:13:58.607813 20806 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790384993574762-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790384993574762-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790384993574762-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:13:58.611091 20806 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:13:58.612957 20806 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:13:58.613073 20806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:13:58.616909 20806 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41427
I20260926 01:13:58.617008 20873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41427 every 8 connection(s)
I20260926 01:13:58.617267 20806 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
I20260926 01:13:58.618327 20874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:13:58.620620 20874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc35806311774c829a4c7795fe9aaf5f: Bootstrap starting.
I20260926 01:13:58.621162 20874 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bc35806311774c829a4c7795fe9aaf5f: Neither blocks nor log segments found. Creating new log.
I20260926 01:13:58.621424 20874 log.cc:824] T 00000000000000000000000000000000 P bc35806311774c829a4c7795fe9aaf5f: Log is configured to *not* fsync() on all Append() calls
I20260926 01:13:58.622095 20874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc35806311774c829a4c7795fe9aaf5f: No bootstrap required, opened a new log
I20260926 01:13:58.623708 20874 raft_consensus.cc:348] T 00000000000000000000000000000000 P bc35806311774c829a4c7795fe9aaf5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc35806311774c829a4c7795fe9aaf5f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41427 } }
I20260926 01:13:58.623852 20874 raft_consensus.cc:374] T 00000000000000000000000000000000 P bc35806311774c829a4c7795fe9aaf5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:13:58.623883 20874 raft_consensus.cc:729] T 00000000000000000000000000000000 P bc35806311774c829a4c7795fe9aaf5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc35806311774c829a4c7795fe9aaf5f, State: Initialized, Role: FOLLOWER
I20260926 01:13:58.624011 20874 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc35806311774c829a4c7795fe9aaf5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc35806311774c829a4c7795fe9aaf5f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41427 } }
I20260926 01:13:58.624070 20874 raft_consensus.cc:388] T 00000000000000000000000000000000 P bc35806311774c829a4c7795fe9aaf5f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:13:58.624099 20874 raft_consensus.cc:482] T 00000000000000000000000000000000 P bc35806311774c829a4c7795fe9aaf5f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:13:58.624145 20874 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc35806311774c829a4c7795fe9aaf5f [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:13:58.624657 20874 raft_consensus.cc:504] T 00000000000000000000000000000000 P bc35806311774c829a4c7795fe9aaf5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc35806311774c829a4c7795fe9aaf5f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41427 } }
I20260926 01:13:58.624750 20874 leader_election.cc:302] T 00000000000000000000000000000000 P bc35806311774c829a4c7795fe9aaf5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bc35806311774c829a4c7795fe9aaf5f; no voters:
I20260926 01:13:58.624905 20874 leader_election.cc:288] T 00000000000000000000000000000000 P bc35806311774c829a4c7795fe9aaf5f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:13:58.624981 20877 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bc35806311774c829a4c7795fe9aaf5f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:13:58.625145 20874 sys_catalog.cc:564] T 00000000000000000000000000000000 P bc35806311774c829a4c7795fe9aaf5f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:13:58.625120 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 20806
I20260926 01:13:58.625264 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790384993574762-392-0/minicluster-data/master-0/wal/instance
I20260926 01:13:58.625420 20877 raft_consensus.cc:686] T 00000000000000000000000000000000 P bc35806311774c829a4c7795fe9aaf5f [term 1 LEADER]: Becoming Leader. State: Replica: bc35806311774c829a4c7795fe9aaf5f, State: Running, Role: LEADER
I20260926 01:13:58.625449 392 external_mini_cluster.cc:1428] Setting key ac13edf8a4fafa97a42cc26d9fa18a99
I20260926 01:13:58.625581 20877 consensus_queue.cc:237] T 00000000000000000000000000000000 P bc35806311774c829a4c7795fe9aaf5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc35806311774c829a4c7795fe9aaf5f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41427 } }
I20260926 01:13:58.626106 20879 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc35806311774c829a4c7795fe9aaf5f [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc35806311774c829a4c7795fe9aaf5f. Latest consensus state: current_term: 1 leader_uuid: "bc35806311774c829a4c7795fe9aaf5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc35806311774c829a4c7795fe9aaf5f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41427 } } }
I20260926 01:13:58.626240 20879 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc35806311774c829a4c7795fe9aaf5f [sys.catalog]: This master's current role is: LEADER
I20260926 01:13:58.626247 20878 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc35806311774c829a4c7795fe9aaf5f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bc35806311774c829a4c7795fe9aaf5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc35806311774c829a4c7795fe9aaf5f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41427 } } }
I20260926 01:13:58.626364 20878 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc35806311774c829a4c7795fe9aaf5f [sys.catalog]: This master's current role is: LEADER
I20260926 01:13:58.626593 20890 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:13:58.627197 20890 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:13:58.628736 20887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:58.625984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58539 (local address 127.0.0.1:41427)
0926 01:13:58.626729 (+ 745us) server_negotiation.cc:206] Beginning negotiation
0926 01:13:58.626741 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 01:13:58.626843 (+ 102us) server_negotiation.cc:407] Connection header received
0926 01:13:58.626966 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:13:58.626973 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:13:58.627006 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:13:58.627155 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:13:58.628210 (+ 1055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:13:58.628214 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:13:58.628214 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:13:58.628218 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:13:58.628277 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:13:58.628291 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:13:58.628351 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 01:13:58.628508 (+ 157us) server_negotiation.cc:299] Negotiation successful
0926 01:13:58.628548 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":688,"thread_start_us":61,"threads_started":1}
I20260926 01:13:58.628773 20890 catalog_manager.cc:1347] Generated new cluster ID: f82862629cae472a9b8fc5e45b56218d
I20260926 01:13:58.629019 20890 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 01:13:58.629637 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 01:13:58.632249 20887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:58.631266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43861 (local address 127.0.0.1:41427)
0926 01:13:58.631303 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 01:13:58.631304 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 01:13:58.631346 (+ 42us) server_negotiation.cc:407] Connection header received
0926 01:13:58.631404 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:13:58.631405 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:13:58.631409 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:13:58.631460 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:13:58.632043 (+ 583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:13:58.632044 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:13:58.632044 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:13:58.632049 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:13:58.632065 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:13:58.632070 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:13:58.632128 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 01:13:58.632170 (+ 42us) server_negotiation.cc:299] Negotiation successful
0926 01:13:58.632183 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 01:13:58.644117 20890 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:13:58.644320 20890 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:13:58.650436 20890 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bc35806311774c829a4c7795fe9aaf5f: Generated new TSK 0
I20260926 01:13:58.650646 20890 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 01:13:58.657624 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 20806
2026-09-26T01:13:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1223 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790384993574762-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 01:13:58 dist-test-slave-7b22 krb5kdc[20905](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 01:13:58 dist-test-slave-7b22 krb5kdc[20905](info): set up 2 sockets
Sep 26 01:13:58 dist-test-slave-7b22 krb5kdc[20905](info): commencing operation
krb5kdc: starting...
W20260926 01:13:59.597679 392 mini_kdc.cc:121] Time spent starting KDC: real 0.923s user 0.001s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790384993574762-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790384993574762-392-0/krb5kdc/test-user.keytab.
Sep 26 01:13:59 dist-test-slave-7b22 krb5kdc[20905](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385239, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T01:13:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:13:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790384993574762-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790384993574762-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 01:13:59.717371 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790384993574762-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790384993574762-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790384993574762-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35033
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790384993574762-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35033
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:13:59.798628 20918 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:13:59.798861 20918 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:13:59.798897 20918 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:13:59.800087 20918 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:13:59.800129 20918 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:13:59.800148 20918 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:13:59.800165 20918 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:13:59.800180 20918 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:13:59.801647 20918 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790384993574762-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790384993574762-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35033
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790384993574762-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35033
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20918
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790384993574762-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:13:59.801936 20918 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:13:59.802145 20918 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:13:59.806985 20923 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:13:59.807087 20918 server_base.cc:1034] running on GCE node
W20260926 01:13:59.806969 20926 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:13:59.807031 20924 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:13:59.807463 20918 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:13:59.807772 20918 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:13:59.808917 20918 hybrid_clock.cc:648] HybridClock initialized: now 1790385239808897 us; error 33 us; skew 500 ppm
Sep 26 01:13:59 dist-test-slave-7b22 krb5kdc[20905](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385239, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:13:59.811333 20918 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 01:13:59.811785 20918 webserver.cc:460] Webserver started at http://127.0.0.1:39873/ using document root <none> and password file <none>
I20260926 01:13:59.812013 20918 fs_manager.cc:359] Metadata directory not provided
I20260926 01:13:59.812060 20918 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:13:59.812182 20918 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:13:59.813231 20918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790384993574762-392-0/minicluster-data/master-0/data/instance:
uuid: "0599ef7b41f5402aa1af64cd850c9339"
format_stamp: "Formatted at 2026-09-26 01:13:59 on dist-test-slave-7b22"
server_key: "70d5e04638534391b64867b1de257d79"
server_key_iv: "690573ad08f881f660883e4aa2b03d21"
server_key_version: "encryptionkey@0"
I20260926 01:13:59.813558 20918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790384993574762-392-0/minicluster-data/master-0/wal/instance:
uuid: "0599ef7b41f5402aa1af64cd850c9339"
format_stamp: "Formatted at 2026-09-26 01:13:59 on dist-test-slave-7b22"
server_key: "70d5e04638534391b64867b1de257d79"
server_key_iv: "690573ad08f881f660883e4aa2b03d21"
server_key_version: "encryptionkey@0"
I20260926 01:13:59.814857 20918 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 01:13:59.815620 20933 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:13:59.815796 20918 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:13:59.815888 20918 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790384993574762-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790384993574762-392-0/minicluster-data/master-0/wal
uuid: "0599ef7b41f5402aa1af64cd850c9339"
format_stamp: "Formatted at 2026-09-26 01:13:59 on dist-test-slave-7b22"
server_key: "70d5e04638534391b64867b1de257d79"
server_key_iv: "690573ad08f881f660883e4aa2b03d21"
server_key_version: "encryptionkey@0"
I20260926 01:13:59.815961 20918 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790384993574762-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790384993574762-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790384993574762-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:13:59.827831 20918 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:13:59.829965 20918 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:13:59.830112 20918 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:13:59.833782 20918 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35033
I20260926 01:13:59.833822 20985 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35033 every 8 connection(s)
I20260926 01:13:59.834221 20918 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
I20260926 01:13:59.835048 20986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:13:59.837311 20986 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0599ef7b41f5402aa1af64cd850c9339: Bootstrap starting.
I20260926 01:13:59.837916 20986 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0599ef7b41f5402aa1af64cd850c9339: Neither blocks nor log segments found. Creating new log.
I20260926 01:13:59.838168 20986 log.cc:824] T 00000000000000000000000000000000 P 0599ef7b41f5402aa1af64cd850c9339: Log is configured to *not* fsync() on all Append() calls
I20260926 01:13:59.838893 20986 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0599ef7b41f5402aa1af64cd850c9339: No bootstrap required, opened a new log
I20260926 01:13:59.840519 20986 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0599ef7b41f5402aa1af64cd850c9339 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0599ef7b41f5402aa1af64cd850c9339" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35033 } }
I20260926 01:13:59.840667 20986 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0599ef7b41f5402aa1af64cd850c9339 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:13:59.840687 20986 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0599ef7b41f5402aa1af64cd850c9339 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0599ef7b41f5402aa1af64cd850c9339, State: Initialized, Role: FOLLOWER
I20260926 01:13:59.840824 20986 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0599ef7b41f5402aa1af64cd850c9339 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0599ef7b41f5402aa1af64cd850c9339" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35033 } }
I20260926 01:13:59.840886 20986 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0599ef7b41f5402aa1af64cd850c9339 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:13:59.840905 20986 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0599ef7b41f5402aa1af64cd850c9339 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:13:59.840950 20986 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0599ef7b41f5402aa1af64cd850c9339 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:13:59.841502 20986 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0599ef7b41f5402aa1af64cd850c9339 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0599ef7b41f5402aa1af64cd850c9339" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35033 } }
I20260926 01:13:59.841596 20986 leader_election.cc:302] T 00000000000000000000000000000000 P 0599ef7b41f5402aa1af64cd850c9339 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0599ef7b41f5402aa1af64cd850c9339; no voters:
I20260926 01:13:59.841764 20986 leader_election.cc:288] T 00000000000000000000000000000000 P 0599ef7b41f5402aa1af64cd850c9339 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:13:59.841873 20989 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0599ef7b41f5402aa1af64cd850c9339 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:13:59.842023 20986 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0599ef7b41f5402aa1af64cd850c9339 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:13:59.842046 20989 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0599ef7b41f5402aa1af64cd850c9339 [term 1 LEADER]: Becoming Leader. State: Replica: 0599ef7b41f5402aa1af64cd850c9339, State: Running, Role: LEADER
I20260926 01:13:59.842134 20989 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0599ef7b41f5402aa1af64cd850c9339 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0599ef7b41f5402aa1af64cd850c9339" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35033 } }
I20260926 01:13:59.842422 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 20918
I20260926 01:13:59.842490 20990 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0599ef7b41f5402aa1af64cd850c9339 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0599ef7b41f5402aa1af64cd850c9339" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0599ef7b41f5402aa1af64cd850c9339" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35033 } } }
I20260926 01:13:59.842546 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790384993574762-392-0/minicluster-data/master-0/wal/instance
I20260926 01:13:59.842556 20990 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0599ef7b41f5402aa1af64cd850c9339 [sys.catalog]: This master's current role is: LEADER
I20260926 01:13:59.842703 20991 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0599ef7b41f5402aa1af64cd850c9339 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0599ef7b41f5402aa1af64cd850c9339. Latest consensus state: current_term: 1 leader_uuid: "0599ef7b41f5402aa1af64cd850c9339" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0599ef7b41f5402aa1af64cd850c9339" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35033 } } }
I20260926 01:13:59.842736 392 external_mini_cluster.cc:1428] Setting key 5affca6c127969bb9c624d9bf40f5753
I20260926 01:13:59.842754 20991 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0599ef7b41f5402aa1af64cd850c9339 [sys.catalog]: This master's current role is: LEADER
I20260926 01:13:59.843102 20993 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:13:59.843669 20993 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:13:59.845233 20993 catalog_manager.cc:1347] Generated new cluster ID: 2c1a1fc27b234191b076021fedfd9210
I20260926 01:13:59.845278 20993 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 01:13:59 dist-test-slave-7b22 krb5kdc[20905](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385239, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 01:13:59.854606 21004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:59.843365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57573 (local address 127.0.0.1:35033)
0926 01:13:59.843839 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0926 01:13:59.843848 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 01:13:59.843878 (+ 30us) server_negotiation.cc:407] Connection header received
0926 01:13:59.843995 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:13:59.844002 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:13:59.844055 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:13:59.844202 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:13:59.845073 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:59.846482 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:59.847529 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:59.848173 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:59.848287 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:59.851029 (+ 2742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:13:59.851033 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:13:59.851034 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:13:59.851047 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:13:59.852942 (+ 1895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:13:59.853361 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:13:59.853363 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:13:59.853364 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:13:59.853465 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:13:59.853642 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:13:59.853642 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:13:59.853643 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:13:59.853967 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:13:59.854114 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0926 01:13:59.854236 (+ 122us) server_negotiation.cc:299] Negotiation successful
0926 01:13:59.854367 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":58,"threads_started":1}
I20260926 01:13:59.855532 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 01:13:59.864387 21004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:13:59.857216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41041 (local address 127.0.0.1:35033)
0926 01:13:59.857303 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0926 01:13:59.857305 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:13:59.857362 (+ 57us) server_negotiation.cc:407] Connection header received
0926 01:13:59.857430 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:13:59.857431 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:13:59.857441 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:13:59.857492 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:13:59.858172 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:59.858945 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:59.859964 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:13:59.860279 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:13:59.860331 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:13:59.861739 (+ 1408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:13:59.861741 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:13:59.861741 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:13:59.861746 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:13:59.863242 (+ 1496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:13:59.863605 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:13:59.863606 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:13:59.863606 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:13:59.863654 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:13:59.863820 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:13:59.863821 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:13:59.863821 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:13:59.863933 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:13:59.864013 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 01:13:59.864123 (+ 110us) server_negotiation.cc:299] Negotiation successful
0926 01:13:59.864200 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 01:13:59.871177 20993 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:13:59.871927 20993 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:13:59.877148 20993 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0599ef7b41f5402aa1af64cd850c9339: Generated new TSK 0
I20260926 01:13:59.877362 20993 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 01:13:59.883843 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 20918
2026-09-26T01:13:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1226 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790384993574762-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 01:13:59 dist-test-slave-7b22 krb5kdc[21017](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 01:13:59 dist-test-slave-7b22 krb5kdc[21017](info): set up 2 sockets
Sep 26 01:13:59 dist-test-slave-7b22 krb5kdc[21017](info): commencing operation
krb5kdc: starting...
W20260926 01:14:00.833729 392 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790384993574762-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790384993574762-392-0/krb5kdc/test-user.keytab.
Sep 26 01:14:00 dist-test-slave-7b22 krb5kdc[21017](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385240, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T01:14:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:14:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790384993574762-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790384993574762-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 01:14:00.951092 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790384993574762-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790384993574762-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790384993574762-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42719
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790384993574762-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42719
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:14:01.031288 21030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:14:01.031509 21030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:14:01.031533 21030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:14:01.032651 21030 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:14:01.032685 21030 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:14:01.032698 21030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:14:01.032708 21030 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:14:01.032719 21030 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:14:01.034192 21030 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790384993574762-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790384993574762-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42719
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790384993574762-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42719
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21030
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790384993574762-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:14:01.034409 21030 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:14:01.034593 21030 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:14:01.039808 21035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:14:01.039860 21036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:14:01.040051 21030 server_base.cc:1034] running on GCE node
W20260926 01:14:01.039808 21038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:14:01.040390 21030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:14:01.040714 21030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:14:01.041929 21030 hybrid_clock.cc:648] HybridClock initialized: now 1790385241041917 us; error 46 us; skew 500 ppm
Sep 26 01:14:01 dist-test-slave-7b22 krb5kdc[21017](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385241, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:14:01.044515 21030 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 01:14:01.044991 21030 webserver.cc:460] Webserver started at http://127.0.0.1:45179/ using document root <none> and password file <none>
I20260926 01:14:01.045225 21030 fs_manager.cc:359] Metadata directory not provided
I20260926 01:14:01.045296 21030 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:14:01.045423 21030 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:14:01.046558 21030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790384993574762-392-0/minicluster-data/master-0/data/instance:
uuid: "4c1c533605a94dda9c90866d1613d410"
format_stamp: "Formatted at 2026-09-26 01:14:01 on dist-test-slave-7b22"
server_key: "693f6ce7934a6aeb4e2fb36b4b8aeeff"
server_key_iv: "67534e5997b8d55b801b9116c9bbde27"
server_key_version: "encryptionkey@0"
I20260926 01:14:01.046881 21030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790384993574762-392-0/minicluster-data/master-0/wal/instance:
uuid: "4c1c533605a94dda9c90866d1613d410"
format_stamp: "Formatted at 2026-09-26 01:14:01 on dist-test-slave-7b22"
server_key: "693f6ce7934a6aeb4e2fb36b4b8aeeff"
server_key_iv: "67534e5997b8d55b801b9116c9bbde27"
server_key_version: "encryptionkey@0"
I20260926 01:14:01.048089 21030 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 01:14:01.048774 21045 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:14:01.048969 21030 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 01:14:01.049063 21030 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790384993574762-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790384993574762-392-0/minicluster-data/master-0/wal
uuid: "4c1c533605a94dda9c90866d1613d410"
format_stamp: "Formatted at 2026-09-26 01:14:01 on dist-test-slave-7b22"
server_key: "693f6ce7934a6aeb4e2fb36b4b8aeeff"
server_key_iv: "67534e5997b8d55b801b9116c9bbde27"
server_key_version: "encryptionkey@0"
I20260926 01:14:01.049141 21030 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790384993574762-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790384993574762-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790384993574762-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:14:01.057770 21030 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:14:01.059544 21030 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:14:01.059695 21030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:14:01.063822 21030 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42719
I20260926 01:14:01.063858 21097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42719 every 8 connection(s)
I20260926 01:14:01.064211 21030 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
I20260926 01:14:01.065050 21098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:14:01.065771 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 21030
I20260926 01:14:01.065910 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790384993574762-392-0/minicluster-data/master-0/wal/instance
I20260926 01:14:01.066089 392 external_mini_cluster.cc:1428] Setting key 431546cdb96040c16405994161a0c4d5
I20260926 01:14:01.067734 21098 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c1c533605a94dda9c90866d1613d410: Bootstrap starting.
I20260926 01:14:01.068415 21098 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c1c533605a94dda9c90866d1613d410: Neither blocks nor log segments found. Creating new log.
I20260926 01:14:01.068718 21098 log.cc:824] T 00000000000000000000000000000000 P 4c1c533605a94dda9c90866d1613d410: Log is configured to *not* fsync() on all Append() calls
I20260926 01:14:01.069432 21098 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c1c533605a94dda9c90866d1613d410: No bootstrap required, opened a new log
I20260926 01:14:01.071703 21098 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c1c533605a94dda9c90866d1613d410 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c1c533605a94dda9c90866d1613d410" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42719 } }
I20260926 01:14:01.071893 21098 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4c1c533605a94dda9c90866d1613d410 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:14:01.071933 21098 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c1c533605a94dda9c90866d1613d410 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c1c533605a94dda9c90866d1613d410, State: Initialized, Role: FOLLOWER
I20260926 01:14:01.072086 21098 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c1c533605a94dda9c90866d1613d410 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c1c533605a94dda9c90866d1613d410" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42719 } }
I20260926 01:14:01.072144 21101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:14:01.066604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52675 (local address 127.0.0.1:42719)
0926 01:14:01.066800 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 01:14:01.066809 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 01:14:01.066841 (+ 32us) server_negotiation.cc:407] Connection header received
0926 01:14:01.067456 (+ 615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:14:01.067465 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:14:01.067525 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:14:01.067677 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:14:01.067905 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:14:01.069182 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:14:01.070046 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:14:01.070716 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:14:01.070823 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:14:01.071491 (+ 668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:14:01.071495 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:14:01.071496 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:14:01.071695 (+ 199us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:14:01.071734 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:14:01.071749 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:14:01.071816 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 01:14:01.071879 (+ 63us) server_negotiation.cc:299] Negotiation successful
0926 01:14:01.071947 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":96,"threads_started":1}
I20260926 01:14:01.072170 21098 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4c1c533605a94dda9c90866d1613d410 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:14:01.072201 21098 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4c1c533605a94dda9c90866d1613d410 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:14:01.072238 21098 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c1c533605a94dda9c90866d1613d410 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:14:01.072839 21098 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4c1c533605a94dda9c90866d1613d410 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c1c533605a94dda9c90866d1613d410" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42719 } }
I20260926 01:14:01.072943 21098 leader_election.cc:302] T 00000000000000000000000000000000 P 4c1c533605a94dda9c90866d1613d410 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4c1c533605a94dda9c90866d1613d410; no voters:
I20260926 01:14:01.073103 21098 leader_election.cc:288] T 00000000000000000000000000000000 P 4c1c533605a94dda9c90866d1613d410 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:14:01.073127 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 01:14:01.073138 21103 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4c1c533605a94dda9c90866d1613d410 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:14:01.073273 21103 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4c1c533605a94dda9c90866d1613d410 [term 1 LEADER]: Becoming Leader. State: Replica: 4c1c533605a94dda9c90866d1613d410, State: Running, Role: LEADER
I20260926 01:14:01.073348 21103 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c1c533605a94dda9c90866d1613d410 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c1c533605a94dda9c90866d1613d410" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42719 } }
I20260926 01:14:01.073346 21098 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c1c533605a94dda9c90866d1613d410 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:14:01.073685 21103 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c1c533605a94dda9c90866d1613d410 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c1c533605a94dda9c90866d1613d410. Latest consensus state: current_term: 1 leader_uuid: "4c1c533605a94dda9c90866d1613d410" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c1c533605a94dda9c90866d1613d410" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42719 } } }
I20260926 01:14:01.073764 21104 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c1c533605a94dda9c90866d1613d410 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4c1c533605a94dda9c90866d1613d410" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c1c533605a94dda9c90866d1613d410" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42719 } } }
I20260926 01:14:01.073843 21103 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c1c533605a94dda9c90866d1613d410 [sys.catalog]: This master's current role is: LEADER
I20260926 01:14:01.073876 21104 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c1c533605a94dda9c90866d1613d410 [sys.catalog]: This master's current role is: LEADER
I20260926 01:14:01.074640 21109 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:14:01.075219 21109 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:14:01.076848 21109 catalog_manager.cc:1347] Generated new cluster ID: 70703160e8254cdc8e44ee51e9ec14f8
I20260926 01:14:01.076916 21109 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 01:14:01.078217 21101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:14:01.074959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53385 (local address 127.0.0.1:42719)
0926 01:14:01.074996 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 01:14:01.074998 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:14:01.075093 (+ 95us) server_negotiation.cc:407] Connection header received
0926 01:14:01.075114 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:14:01.075114 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:14:01.075124 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:14:01.075172 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:14:01.075351 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:14:01.075992 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:14:01.076907 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:14:01.077183 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:14:01.077223 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:14:01.077991 (+ 768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:14:01.077992 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:14:01.077993 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:14:01.077997 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:14:01.078011 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:14:01.078015 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:14:01.078037 (+ 22us) server_negotiation.cc:1035] Waiting for connection context
0926 01:14:01.078120 (+ 83us) server_negotiation.cc:299] Negotiation successful
0926 01:14:01.078143 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 01:14:01.088461 21109 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:14:01.089005 21109 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:14:01.099272 21109 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4c1c533605a94dda9c90866d1613d410: Generated new TSK 0
I20260926 01:14:01.099478 21109 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 01:14:01.104671 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 21030
2026-09-26T01:14:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1220 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790384993574762-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 01:14:01 dist-test-slave-7b22 krb5kdc[21129](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 01:14:01 dist-test-slave-7b22 krb5kdc[21129](info): set up 2 sockets
Sep 26 01:14:01 dist-test-slave-7b22 krb5kdc[21129](info): commencing operation
krb5kdc: starting...
W20260926 01:14:02.057925 392 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790384993574762-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790384993574762-392-0/krb5kdc/test-user.keytab.
Sep 26 01:14:02 dist-test-slave-7b22 krb5kdc[21129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385242, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T01:14:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:14:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790384993574762-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790384993574762-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 01:14:02.177294 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790384993574762-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790384993574762-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790384993574762-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45595
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790384993574762-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45595
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:14:02.259685 21142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:14:02.259935 21142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:14:02.259959 21142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:14:02.261191 21142 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:14:02.261234 21142 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:14:02.261247 21142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:14:02.261258 21142 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:14:02.261268 21142 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:14:02.262948 21142 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790384993574762-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790384993574762-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45595
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790384993574762-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45595
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21142
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790384993574762-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:14:02.263187 21142 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:14:02.263368 21142 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:14:02.268579 21147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:14:02.268589 21150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:14:02.268633 21142 server_base.cc:1034] running on GCE node
W20260926 01:14:02.268553 21148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:14:02.269007 21142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:14:02.269325 21142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:14:02.270496 21142 hybrid_clock.cc:648] HybridClock initialized: now 1790385242270476 us; error 31 us; skew 500 ppm
Sep 26 01:14:02 dist-test-slave-7b22 krb5kdc[21129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385242, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:14:02.272990 21142 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 01:14:02.273447 21142 webserver.cc:460] Webserver started at http://127.0.0.1:33671/ using document root <none> and password file <none>
I20260926 01:14:02.273645 21142 fs_manager.cc:359] Metadata directory not provided
I20260926 01:14:02.273695 21142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:14:02.273846 21142 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:14:02.274904 21142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790384993574762-392-0/minicluster-data/master-0/data/instance:
uuid: "80cdd5c709a045dabc5b101d578a422e"
format_stamp: "Formatted at 2026-09-26 01:14:02 on dist-test-slave-7b22"
server_key: "c4800b103a638dd5a8f5f052321871d3"
server_key_iv: "dd67a897e6e091ba9fe28058f36010e5"
server_key_version: "encryptionkey@0"
I20260926 01:14:02.275238 21142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790384993574762-392-0/minicluster-data/master-0/wal/instance:
uuid: "80cdd5c709a045dabc5b101d578a422e"
format_stamp: "Formatted at 2026-09-26 01:14:02 on dist-test-slave-7b22"
server_key: "c4800b103a638dd5a8f5f052321871d3"
server_key_iv: "dd67a897e6e091ba9fe28058f36010e5"
server_key_version: "encryptionkey@0"
I20260926 01:14:02.276471 21142 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 01:14:02.277177 21157 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:14:02.277359 21142 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 01:14:02.277452 21142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790384993574762-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790384993574762-392-0/minicluster-data/master-0/wal
uuid: "80cdd5c709a045dabc5b101d578a422e"
format_stamp: "Formatted at 2026-09-26 01:14:02 on dist-test-slave-7b22"
server_key: "c4800b103a638dd5a8f5f052321871d3"
server_key_iv: "dd67a897e6e091ba9fe28058f36010e5"
server_key_version: "encryptionkey@0"
I20260926 01:14:02.277518 21142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790384993574762-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790384993574762-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790384993574762-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:14:02.280893 21142 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:14:02.282903 21142 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:14:02.283042 21142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:14:02.286684 21142 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45595
I20260926 01:14:02.286715 21209 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45595 every 8 connection(s)
I20260926 01:14:02.287035 21142 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
I20260926 01:14:02.287978 21210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:14:02.290292 21210 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80cdd5c709a045dabc5b101d578a422e: Bootstrap starting.
I20260926 01:14:02.290839 21210 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 80cdd5c709a045dabc5b101d578a422e: Neither blocks nor log segments found. Creating new log.
I20260926 01:14:02.291095 21210 log.cc:824] T 00000000000000000000000000000000 P 80cdd5c709a045dabc5b101d578a422e: Log is configured to *not* fsync() on all Append() calls
I20260926 01:14:02.291777 21210 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80cdd5c709a045dabc5b101d578a422e: No bootstrap required, opened a new log
I20260926 01:14:02.292127 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 21142
I20260926 01:14:02.292253 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790384993574762-392-0/minicluster-data/master-0/wal/instance
I20260926 01:14:02.292402 392 external_mini_cluster.cc:1428] Setting key eeaa213a1049a7ff82dfda7818325bf9
I20260926 01:14:02.293807 21210 raft_consensus.cc:348] T 00000000000000000000000000000000 P 80cdd5c709a045dabc5b101d578a422e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80cdd5c709a045dabc5b101d578a422e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45595 } }
I20260926 01:14:02.294032 21210 raft_consensus.cc:374] T 00000000000000000000000000000000 P 80cdd5c709a045dabc5b101d578a422e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:14:02.294068 21210 raft_consensus.cc:729] T 00000000000000000000000000000000 P 80cdd5c709a045dabc5b101d578a422e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80cdd5c709a045dabc5b101d578a422e, State: Initialized, Role: FOLLOWER
I20260926 01:14:02.294211 21210 consensus_queue.cc:260] T 00000000000000000000000000000000 P 80cdd5c709a045dabc5b101d578a422e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80cdd5c709a045dabc5b101d578a422e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45595 } }
I20260926 01:14:02.294309 21210 raft_consensus.cc:388] T 00000000000000000000000000000000 P 80cdd5c709a045dabc5b101d578a422e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:14:02.294337 21210 raft_consensus.cc:482] T 00000000000000000000000000000000 P 80cdd5c709a045dabc5b101d578a422e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:14:02.294366 21210 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 80cdd5c709a045dabc5b101d578a422e [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:14:02.295396 21214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:14:02.292918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59831 (local address 127.0.0.1:45595)
0926 01:14:02.293101 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 01:14:02.293111 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 01:14:02.293162 (+ 51us) server_negotiation.cc:407] Connection header received
0926 01:14:02.293762 (+ 600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:14:02.293771 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:14:02.293847 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:14:02.294025 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:14:02.294883 (+ 858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:14:02.294888 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:14:02.294888 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:14:02.294893 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:14:02.294952 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:14:02.294969 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:14:02.295048 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 01:14:02.295144 (+ 96us) server_negotiation.cc:299] Negotiation successful
0926 01:14:02.295187 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":68,"threads_started":1}
I20260926 01:14:02.295466 21210 raft_consensus.cc:504] T 00000000000000000000000000000000 P 80cdd5c709a045dabc5b101d578a422e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80cdd5c709a045dabc5b101d578a422e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45595 } }
I20260926 01:14:02.295553 21210 leader_election.cc:302] T 00000000000000000000000000000000 P 80cdd5c709a045dabc5b101d578a422e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 80cdd5c709a045dabc5b101d578a422e; no voters:
I20260926 01:14:02.295709 21210 leader_election.cc:288] T 00000000000000000000000000000000 P 80cdd5c709a045dabc5b101d578a422e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:14:02.295737 21215 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 80cdd5c709a045dabc5b101d578a422e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:14:02.295832 21215 raft_consensus.cc:686] T 00000000000000000000000000000000 P 80cdd5c709a045dabc5b101d578a422e [term 1 LEADER]: Becoming Leader. State: Replica: 80cdd5c709a045dabc5b101d578a422e, State: Running, Role: LEADER
I20260926 01:14:02.295981 21210 sys_catalog.cc:564] T 00000000000000000000000000000000 P 80cdd5c709a045dabc5b101d578a422e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:14:02.295965 21215 consensus_queue.cc:237] T 00000000000000000000000000000000 P 80cdd5c709a045dabc5b101d578a422e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80cdd5c709a045dabc5b101d578a422e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45595 } }
I20260926 01:14:02.296419 21216 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80cdd5c709a045dabc5b101d578a422e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 80cdd5c709a045dabc5b101d578a422e. Latest consensus state: current_term: 1 leader_uuid: "80cdd5c709a045dabc5b101d578a422e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80cdd5c709a045dabc5b101d578a422e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45595 } } }
I20260926 01:14:02.296500 21216 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80cdd5c709a045dabc5b101d578a422e [sys.catalog]: This master's current role is: LEADER
I20260926 01:14:02.296538 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 01:14:02.296701 21217 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80cdd5c709a045dabc5b101d578a422e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "80cdd5c709a045dabc5b101d578a422e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80cdd5c709a045dabc5b101d578a422e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45595 } } }
I20260926 01:14:02.297020 21217 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80cdd5c709a045dabc5b101d578a422e [sys.catalog]: This master's current role is: LEADER
I20260926 01:14:02.297202 21223 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:14:02.297689 21223 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:14:02.299611 21214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:14:02.298211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50061 (local address 127.0.0.1:45595)
0926 01:14:02.298253 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 01:14:02.298255 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:14:02.298499 (+ 244us) server_negotiation.cc:407] Connection header received
0926 01:14:02.298565 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:14:02.298565 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:14:02.298570 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:14:02.298639 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:14:02.299367 (+ 728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:14:02.299369 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:14:02.299369 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:14:02.299374 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:14:02.299391 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:14:02.299395 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:14:02.299452 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 01:14:02.299533 (+ 81us) server_negotiation.cc:299] Negotiation successful
0926 01:14:02.299546 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 01:14:02.300153 21223 catalog_manager.cc:1347] Generated new cluster ID: 946c2cf61ae24b588978cb3b66f64ad5
I20260926 01:14:02.300194 21223 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 01:14:02.306500 21223 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:14:02.306674 21223 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:14:02.310253 21223 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 80cdd5c709a045dabc5b101d578a422e: Generated new TSK 0
I20260926 01:14:02.310451 21223 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 01:14:02.312983 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 21142
2026-09-26T01:14:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1208 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790384993574762-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 01:14:02 dist-test-slave-7b22 krb5kdc[21241](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 01:14:02 dist-test-slave-7b22 krb5kdc[21241](info): set up 2 sockets
Sep 26 01:14:02 dist-test-slave-7b22 krb5kdc[21241](info): commencing operation
krb5kdc: starting...
W20260926 01:14:03.273733 392 mini_kdc.cc:121] Time spent starting KDC: real 0.943s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790384993574762-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790384993574762-392-0/krb5kdc/test-user.keytab.
Sep 26 01:14:03 dist-test-slave-7b22 krb5kdc[21241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385243, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T01:14:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:14:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790384993574762-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790384993574762-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 01:14:03.389662 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790384993574762-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790384993574762-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790384993574762-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36945
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790384993574762-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36945
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:14:03.469317 21254 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:14:03.469579 21254 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:14:03.469614 21254 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:14:03.470881 21254 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:14:03.470928 21254 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:14:03.470947 21254 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:14:03.470963 21254 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:14:03.470978 21254 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:14:03.472473 21254 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790384993574762-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790384993574762-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36945
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790384993574762-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36945
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21254
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790384993574762-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:14:03.472723 21254 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:14:03.472929 21254 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:14:03.477612 21259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:14:03.477623 21260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:14:03.477778 21254 server_base.cc:1034] running on GCE node
W20260926 01:14:03.477627 21262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:14:03.478082 21254 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:14:03.478332 21254 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:14:03.479481 21254 hybrid_clock.cc:648] HybridClock initialized: now 1790385243479452 us; error 40 us; skew 500 ppm
Sep 26 01:14:03 dist-test-slave-7b22 krb5kdc[21241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385243, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:14:03.487152 21254 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 01:14:03.487603 21254 webserver.cc:460] Webserver started at http://127.0.0.1:45865/ using document root <none> and password file <none>
I20260926 01:14:03.487875 21254 fs_manager.cc:359] Metadata directory not provided
I20260926 01:14:03.487932 21254 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:14:03.488065 21254 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:14:03.489159 21254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790384993574762-392-0/minicluster-data/master-0/data/instance:
uuid: "13cd623fc4034359bf2d75eecfe6ab60"
format_stamp: "Formatted at 2026-09-26 01:14:03 on dist-test-slave-7b22"
server_key: "73159d4c1611aba21f1921797f121f2b"
server_key_iv: "4ad08fae45804e8527160288fea94097"
server_key_version: "encryptionkey@0"
I20260926 01:14:03.489498 21254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790384993574762-392-0/minicluster-data/master-0/wal/instance:
uuid: "13cd623fc4034359bf2d75eecfe6ab60"
format_stamp: "Formatted at 2026-09-26 01:14:03 on dist-test-slave-7b22"
server_key: "73159d4c1611aba21f1921797f121f2b"
server_key_iv: "4ad08fae45804e8527160288fea94097"
server_key_version: "encryptionkey@0"
I20260926 01:14:03.490899 21254 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 01:14:03.491591 21269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:14:03.491743 21254 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 01:14:03.491827 21254 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790384993574762-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790384993574762-392-0/minicluster-data/master-0/wal
uuid: "13cd623fc4034359bf2d75eecfe6ab60"
format_stamp: "Formatted at 2026-09-26 01:14:03 on dist-test-slave-7b22"
server_key: "73159d4c1611aba21f1921797f121f2b"
server_key_iv: "4ad08fae45804e8527160288fea94097"
server_key_version: "encryptionkey@0"
I20260926 01:14:03.491887 21254 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790384993574762-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790384993574762-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790384993574762-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:14:03.511492 21254 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:14:03.513453 21254 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:14:03.513590 21254 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:14:03.517309 21254 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36945
I20260926 01:14:03.517364 21321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36945 every 8 connection(s)
I20260926 01:14:03.517689 21254 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
I20260926 01:14:03.518577 21322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:14:03.520836 21322 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13cd623fc4034359bf2d75eecfe6ab60: Bootstrap starting.
I20260926 01:14:03.521399 21322 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 13cd623fc4034359bf2d75eecfe6ab60: Neither blocks nor log segments found. Creating new log.
I20260926 01:14:03.521651 21322 log.cc:824] T 00000000000000000000000000000000 P 13cd623fc4034359bf2d75eecfe6ab60: Log is configured to *not* fsync() on all Append() calls
I20260926 01:14:03.522367 21322 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13cd623fc4034359bf2d75eecfe6ab60: No bootstrap required, opened a new log
I20260926 01:14:03.523957 21322 raft_consensus.cc:348] T 00000000000000000000000000000000 P 13cd623fc4034359bf2d75eecfe6ab60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13cd623fc4034359bf2d75eecfe6ab60" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36945 } }
I20260926 01:14:03.524106 21322 raft_consensus.cc:374] T 00000000000000000000000000000000 P 13cd623fc4034359bf2d75eecfe6ab60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:14:03.524128 21322 raft_consensus.cc:729] T 00000000000000000000000000000000 P 13cd623fc4034359bf2d75eecfe6ab60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13cd623fc4034359bf2d75eecfe6ab60, State: Initialized, Role: FOLLOWER
I20260926 01:14:03.524271 21322 consensus_queue.cc:260] T 00000000000000000000000000000000 P 13cd623fc4034359bf2d75eecfe6ab60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13cd623fc4034359bf2d75eecfe6ab60" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36945 } }
I20260926 01:14:03.524338 21322 raft_consensus.cc:388] T 00000000000000000000000000000000 P 13cd623fc4034359bf2d75eecfe6ab60 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:14:03.524356 21322 raft_consensus.cc:482] T 00000000000000000000000000000000 P 13cd623fc4034359bf2d75eecfe6ab60 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:14:03.524386 21322 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 13cd623fc4034359bf2d75eecfe6ab60 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:14:03.524861 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 21254
I20260926 01:14:03.524924 21322 raft_consensus.cc:504] T 00000000000000000000000000000000 P 13cd623fc4034359bf2d75eecfe6ab60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13cd623fc4034359bf2d75eecfe6ab60" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36945 } }
I20260926 01:14:03.524973 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790384993574762-392-0/minicluster-data/master-0/wal/instance
I20260926 01:14:03.525013 21322 leader_election.cc:302] T 00000000000000000000000000000000 P 13cd623fc4034359bf2d75eecfe6ab60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 13cd623fc4034359bf2d75eecfe6ab60; no voters:
I20260926 01:14:03.525164 392 external_mini_cluster.cc:1428] Setting key 593fb7663c3b818835330b5355383501
I20260926 01:14:03.525177 21322 leader_election.cc:288] T 00000000000000000000000000000000 P 13cd623fc4034359bf2d75eecfe6ab60 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:14:03.525225 21325 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 13cd623fc4034359bf2d75eecfe6ab60 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:14:03.525381 21325 raft_consensus.cc:686] T 00000000000000000000000000000000 P 13cd623fc4034359bf2d75eecfe6ab60 [term 1 LEADER]: Becoming Leader. State: Replica: 13cd623fc4034359bf2d75eecfe6ab60, State: Running, Role: LEADER
I20260926 01:14:03.525451 21322 sys_catalog.cc:564] T 00000000000000000000000000000000 P 13cd623fc4034359bf2d75eecfe6ab60 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:14:03.525493 21325 consensus_queue.cc:237] T 00000000000000000000000000000000 P 13cd623fc4034359bf2d75eecfe6ab60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13cd623fc4034359bf2d75eecfe6ab60" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36945 } }
I20260926 01:14:03.525934 21326 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13cd623fc4034359bf2d75eecfe6ab60 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "13cd623fc4034359bf2d75eecfe6ab60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13cd623fc4034359bf2d75eecfe6ab60" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36945 } } }
I20260926 01:14:03.526016 21326 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13cd623fc4034359bf2d75eecfe6ab60 [sys.catalog]: This master's current role is: LEADER
I20260926 01:14:03.526211 21327 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13cd623fc4034359bf2d75eecfe6ab60 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 13cd623fc4034359bf2d75eecfe6ab60. Latest consensus state: current_term: 1 leader_uuid: "13cd623fc4034359bf2d75eecfe6ab60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13cd623fc4034359bf2d75eecfe6ab60" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36945 } } }
I20260926 01:14:03.526494 21327 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13cd623fc4034359bf2d75eecfe6ab60 [sys.catalog]: This master's current role is: LEADER
I20260926 01:14:03.526552 21331 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:14:03.527132 21331 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:14:03.528919 21331 catalog_manager.cc:1347] Generated new cluster ID: 94fad7227a91422495ead579062f0bf4
I20260926 01:14:03.528976 21331 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 01:14:03 dist-test-slave-7b22 krb5kdc[21241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385243, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 01:14:03.536135 21329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:14:03.525668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.100:48669 (local address 127.0.0.1:36945)
0926 01:14:03.525850 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 01:14:03.525860 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 01:14:03.526034 (+ 174us) server_negotiation.cc:407] Connection header received
0926 01:14:03.526256 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:14:03.526265 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:14:03.526337 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:14:03.526461 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:14:03.527357 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:14:03.528547 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:14:03.530418 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:14:03.530834 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:14:03.530934 (+ 100us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:14:03.533124 (+ 2190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:14:03.533128 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:14:03.533128 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:14:03.533141 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:14:03.534661 (+ 1520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:14:03.535037 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:14:03.535039 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:14:03.535040 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:14:03.535138 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:14:03.535336 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:14:03.535337 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:14:03.535337 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:14:03.535584 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:14:03.535744 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0926 01:14:03.535851 (+ 107us) server_negotiation.cc:299] Negotiation successful
0926 01:14:03.535963 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":60,"threads_started":1}
I20260926 01:14:03.537179 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 01:14:03.544231 21329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:14:03.538558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.100:59149 (local address 127.0.0.1:36945)
0926 01:14:03.538596 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 01:14:03.538598 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:14:03.538702 (+ 104us) server_negotiation.cc:407] Connection header received
0926 01:14:03.538758 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:14:03.538760 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:14:03.538770 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:14:03.538822 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:14:03.539325 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:14:03.539846 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:14:03.540513 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:14:03.540741 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:14:03.540795 (+ 54us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:14:03.541957 (+ 1162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:14:03.541959 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:14:03.541959 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:14:03.541966 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:14:03.543192 (+ 1226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:14:03.543535 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:14:03.543536 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:14:03.543537 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:14:03.543587 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:14:03.543768 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:14:03.543769 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:14:03.543769 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:14:03.543881 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:14:03.543981 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 01:14:03.544082 (+ 101us) server_negotiation.cc:299] Negotiation successful
0926 01:14:03.544138 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 01:14:03.553180 21331 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:14:03.553943 21331 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:14:03.556584 21331 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 13cd623fc4034359bf2d75eecfe6ab60: Generated new TSK 0
I20260926 01:14:03.556761 21331 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 01:14:03.561257 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 21254
W20260926 01:14:03.565083 21239 connection.cc:535] client connection to 127.0.0.1:36945 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36945): Connection reset by peer (error 104)
2026-09-26T01:14:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1248 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790384993574762-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 01:14:03 dist-test-slave-7b22 krb5kdc[21353](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 01:14:03 dist-test-slave-7b22 krb5kdc[21353](info): set up 2 sockets
Sep 26 01:14:03 dist-test-slave-7b22 krb5kdc[21353](info): commencing operation
krb5kdc: starting...
W20260926 01:14:04.517695 392 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790384993574762-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790384993574762-392-0/krb5kdc/test-user.keytab.
Sep 26 01:14:04 dist-test-slave-7b22 krb5kdc[21353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385244, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T01:14:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:14:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790384993574762-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790384993574762-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 01:14:04.634927 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790384993574762-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790384993574762-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790384993574762-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34401
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790384993574762-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34401
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:14:04.714077 21366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:14:04.714309 21366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:14:04.714344 21366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:14:04.715552 21366 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:14:04.715596 21366 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:14:04.715615 21366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:14:04.715632 21366 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:14:04.715646 21366 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:14:04.717196 21366 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790384993574762-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790384993574762-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34401
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790384993574762-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34401
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21366
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790384993574762-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:14:04.717459 21366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:14:04.717675 21366 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:14:04.723026 21371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:14:04.723105 21374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:14:04.723075 21366 server_base.cc:1034] running on GCE node
W20260926 01:14:04.723177 21372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:14:04.723529 21366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:14:04.723874 21366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:14:04.725032 21366 hybrid_clock.cc:648] HybridClock initialized: now 1790385244725014 us; error 36 us; skew 500 ppm
Sep 26 01:14:04 dist-test-slave-7b22 krb5kdc[21353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385244, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:14:04.727593 21366 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 01:14:04.728058 21366 webserver.cc:460] Webserver started at http://127.0.0.1:37065/ using document root <none> and password file <none>
I20260926 01:14:04.728276 21366 fs_manager.cc:359] Metadata directory not provided
I20260926 01:14:04.728325 21366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:14:04.728449 21366 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:14:04.729568 21366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790384993574762-392-0/minicluster-data/master-0/data/instance:
uuid: "6d30dc9ff5e649cb84c4d675bea6a1ad"
format_stamp: "Formatted at 2026-09-26 01:14:04 on dist-test-slave-7b22"
server_key: "75af8ad27a90706bf7b0bcd02253166b"
server_key_iv: "3083ed890aac54082ecd1fe014636646"
server_key_version: "encryptionkey@0"
I20260926 01:14:04.729965 21366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790384993574762-392-0/minicluster-data/master-0/wal/instance:
uuid: "6d30dc9ff5e649cb84c4d675bea6a1ad"
format_stamp: "Formatted at 2026-09-26 01:14:04 on dist-test-slave-7b22"
server_key: "75af8ad27a90706bf7b0bcd02253166b"
server_key_iv: "3083ed890aac54082ecd1fe014636646"
server_key_version: "encryptionkey@0"
I20260926 01:14:04.731328 21366 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 01:14:04.732228 21381 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:14:04.732435 21366 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 01:14:04.732527 21366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790384993574762-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790384993574762-392-0/minicluster-data/master-0/wal
uuid: "6d30dc9ff5e649cb84c4d675bea6a1ad"
format_stamp: "Formatted at 2026-09-26 01:14:04 on dist-test-slave-7b22"
server_key: "75af8ad27a90706bf7b0bcd02253166b"
server_key_iv: "3083ed890aac54082ecd1fe014636646"
server_key_version: "encryptionkey@0"
I20260926 01:14:04.732592 21366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790384993574762-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790384993574762-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790384993574762-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:14:04.758558 21366 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:14:04.760524 21366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:14:04.760663 21366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:14:04.764485 21366 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34401
I20260926 01:14:04.764556 21433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34401 every 8 connection(s)
I20260926 01:14:04.764863 21366 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
I20260926 01:14:04.765735 21434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:14:04.768085 21434 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d30dc9ff5e649cb84c4d675bea6a1ad: Bootstrap starting.
I20260926 01:14:04.768667 21434 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6d30dc9ff5e649cb84c4d675bea6a1ad: Neither blocks nor log segments found. Creating new log.
I20260926 01:14:04.768924 21434 log.cc:824] T 00000000000000000000000000000000 P 6d30dc9ff5e649cb84c4d675bea6a1ad: Log is configured to *not* fsync() on all Append() calls
I20260926 01:14:04.769699 21434 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d30dc9ff5e649cb84c4d675bea6a1ad: No bootstrap required, opened a new log
I20260926 01:14:04.769783 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 21366
I20260926 01:14:04.769934 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790384993574762-392-0/minicluster-data/master-0/wal/instance
I20260926 01:14:04.770119 392 external_mini_cluster.cc:1428] Setting key 5f85a0f850ba5a41dd9a96fa08793c41
I20260926 01:14:04.771715 21434 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6d30dc9ff5e649cb84c4d675bea6a1ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d30dc9ff5e649cb84c4d675bea6a1ad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34401 } }
I20260926 01:14:04.771870 21434 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6d30dc9ff5e649cb84c4d675bea6a1ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:14:04.771903 21434 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6d30dc9ff5e649cb84c4d675bea6a1ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d30dc9ff5e649cb84c4d675bea6a1ad, State: Initialized, Role: FOLLOWER
I20260926 01:14:04.772064 21434 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6d30dc9ff5e649cb84c4d675bea6a1ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d30dc9ff5e649cb84c4d675bea6a1ad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34401 } }
I20260926 01:14:04.772161 21434 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6d30dc9ff5e649cb84c4d675bea6a1ad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:14:04.772189 21434 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6d30dc9ff5e649cb84c4d675bea6a1ad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:14:04.772238 21434 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6d30dc9ff5e649cb84c4d675bea6a1ad [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:14:04.772827 21434 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6d30dc9ff5e649cb84c4d675bea6a1ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d30dc9ff5e649cb84c4d675bea6a1ad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34401 } }
I20260926 01:14:04.772924 21434 leader_election.cc:302] T 00000000000000000000000000000000 P 6d30dc9ff5e649cb84c4d675bea6a1ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6d30dc9ff5e649cb84c4d675bea6a1ad; no voters:
I20260926 01:14:04.773087 21434 leader_election.cc:288] T 00000000000000000000000000000000 P 6d30dc9ff5e649cb84c4d675bea6a1ad [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:14:04.773129 21439 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6d30dc9ff5e649cb84c4d675bea6a1ad [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:14:04.773289 21439 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6d30dc9ff5e649cb84c4d675bea6a1ad [term 1 LEADER]: Becoming Leader. State: Replica: 6d30dc9ff5e649cb84c4d675bea6a1ad, State: Running, Role: LEADER
I20260926 01:14:04.773345 21434 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6d30dc9ff5e649cb84c4d675bea6a1ad [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:14:04.773372 21439 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d30dc9ff5e649cb84c4d675bea6a1ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d30dc9ff5e649cb84c4d675bea6a1ad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34401 } }
I20260926 01:14:04.773684 21439 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d30dc9ff5e649cb84c4d675bea6a1ad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d30dc9ff5e649cb84c4d675bea6a1ad. Latest consensus state: current_term: 1 leader_uuid: "6d30dc9ff5e649cb84c4d675bea6a1ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d30dc9ff5e649cb84c4d675bea6a1ad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34401 } } }
I20260926 01:14:04.773751 21439 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d30dc9ff5e649cb84c4d675bea6a1ad [sys.catalog]: This master's current role is: LEADER
I20260926 01:14:04.773967 21444 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:14:04.774621 21444 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:14:04.774816 21440 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d30dc9ff5e649cb84c4d675bea6a1ad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6d30dc9ff5e649cb84c4d675bea6a1ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d30dc9ff5e649cb84c4d675bea6a1ad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34401 } } }
I20260926 01:14:04.774942 21440 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d30dc9ff5e649cb84c4d675bea6a1ad [sys.catalog]: This master's current role is: LEADER
I20260926 01:14:04.776578 21444 catalog_manager.cc:1347] Generated new cluster ID: 8b4ffb9833104903bbc1db8f1e0323eb
I20260926 01:14:04.776631 21444 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 01:14:04.776826 21438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:14:04.770656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.100:42635 (local address 127.0.0.1:34401)
0926 01:14:04.770824 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 01:14:04.770831 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 01:14:04.770857 (+ 26us) server_negotiation.cc:407] Connection header received
0926 01:14:04.771696 (+ 839us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:14:04.771704 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:14:04.771763 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:14:04.771906 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:14:04.772133 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:14:04.773510 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:14:04.774558 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:14:04.775147 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:14:04.775252 (+ 105us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:14:04.776011 (+ 759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:14:04.776015 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:14:04.776015 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:14:04.776211 (+ 196us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:14:04.776254 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:14:04.776271 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:14:04.776404 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0926 01:14:04.776573 (+ 169us) server_negotiation.cc:299] Negotiation successful
0926 01:14:04.776601 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":68,"threads_started":1}
I20260926 01:14:04.777748 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 01:14:04.789554 21438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:14:04.779185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.100:33099 (local address 127.0.0.1:34401)
0926 01:14:04.779215 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0926 01:14:04.779217 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:14:04.786090 (+ 6873us) server_negotiation.cc:407] Connection header received
0926 01:14:04.786154 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:14:04.786155 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:14:04.786169 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:14:04.786236 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:14:04.786395 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:14:04.787165 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:14:04.788127 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:14:04.788484 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:14:04.788550 (+ 66us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:14:04.789277 (+ 727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:14:04.789278 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:14:04.789279 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:14:04.789284 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:14:04.789302 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:14:04.789307 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:14:04.789375 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 01:14:04.789445 (+ 70us) server_negotiation.cc:299] Negotiation successful
0926 01:14:04.789460 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260926 01:14:04.799268 21444 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:14:04.800050 21444 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:14:04.809355 21444 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6d30dc9ff5e649cb84c4d675bea6a1ad: Generated new TSK 0
I20260926 01:14:04.809580 21444 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 01:14:04.815164 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 21366
W20260926 01:14:04.819018 21351 connection.cc:535] client connection to 127.0.0.1:34401 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34401): Connection reset by peer (error 104)
2026-09-26T01:14:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1254 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790384993574762-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 01:14:04 dist-test-slave-7b22 krb5kdc[21465](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 01:14:04 dist-test-slave-7b22 krb5kdc[21465](info): set up 2 sockets
Sep 26 01:14:04 dist-test-slave-7b22 krb5kdc[21465](info): commencing operation
krb5kdc: starting...
W20260926 01:14:05.765679 392 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790384993574762-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790384993574762-392-0/krb5kdc/test-user.keytab.
Sep 26 01:14:05 dist-test-slave-7b22 krb5kdc[21465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385245, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T01:14:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:14:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790384993574762-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790384993574762-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 01:14:05.884709 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790384993574762-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790384993574762-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790384993574762-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45005
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790384993574762-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45005
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:14:05.967595 21478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:14:05.967829 21478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:14:05.967864 21478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:14:05.969137 21478 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:14:05.969192 21478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:14:05.969206 21478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:14:05.969218 21478 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:14:05.969230 21478 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:14:05.970896 21478 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790384993574762-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790384993574762-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45005
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790384993574762-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45005
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21478
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790384993574762-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:14:05.971144 21478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:14:05.971364 21478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:14:05.976126 21484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:14:05.976219 21486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:14:05.976228 21483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:14:05.976366 21478 server_base.cc:1034] running on GCE node
I20260926 01:14:05.976625 21478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:14:05.976929 21478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:14:05.978091 21478 hybrid_clock.cc:648] HybridClock initialized: now 1790385245978071 us; error 31 us; skew 500 ppm
Sep 26 01:14:05 dist-test-slave-7b22 krb5kdc[21465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385245, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:14:05.980502 21478 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 01:14:05.980963 21478 webserver.cc:460] Webserver started at http://127.0.0.1:35491/ using document root <none> and password file <none>
I20260926 01:14:05.981176 21478 fs_manager.cc:359] Metadata directory not provided
I20260926 01:14:05.981225 21478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:14:05.981348 21478 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:14:05.982424 21478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790384993574762-392-0/minicluster-data/master-0/data/instance:
uuid: "a5dbd019d6f94f1f9f1fa4849238022f"
format_stamp: "Formatted at 2026-09-26 01:14:05 on dist-test-slave-7b22"
server_key: "7e4124a1e12997e95a432de7910e17e1"
server_key_iv: "5625671fc89a7cf05dceec3bfbe1f888"
server_key_version: "encryptionkey@0"
I20260926 01:14:05.982841 21478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790384993574762-392-0/minicluster-data/master-0/wal/instance:
uuid: "a5dbd019d6f94f1f9f1fa4849238022f"
format_stamp: "Formatted at 2026-09-26 01:14:05 on dist-test-slave-7b22"
server_key: "7e4124a1e12997e95a432de7910e17e1"
server_key_iv: "5625671fc89a7cf05dceec3bfbe1f888"
server_key_version: "encryptionkey@0"
I20260926 01:14:05.984180 21478 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 01:14:05.984965 21493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:14:05.985145 21478 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:14:05.985231 21478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790384993574762-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790384993574762-392-0/minicluster-data/master-0/wal
uuid: "a5dbd019d6f94f1f9f1fa4849238022f"
format_stamp: "Formatted at 2026-09-26 01:14:05 on dist-test-slave-7b22"
server_key: "7e4124a1e12997e95a432de7910e17e1"
server_key_iv: "5625671fc89a7cf05dceec3bfbe1f888"
server_key_version: "encryptionkey@0"
I20260926 01:14:05.985298 21478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790384993574762-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790384993574762-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790384993574762-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:14:05.988677 21478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:14:05.990849 21478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:14:05.990995 21478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:14:05.994688 21478 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45005
I20260926 01:14:05.994719 21545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45005 every 8 connection(s)
I20260926 01:14:05.995050 21478 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
I20260926 01:14:05.996035 21546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:14:05.998347 21546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5dbd019d6f94f1f9f1fa4849238022f: Bootstrap starting.
I20260926 01:14:05.998883 21546 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a5dbd019d6f94f1f9f1fa4849238022f: Neither blocks nor log segments found. Creating new log.
I20260926 01:14:05.999109 21546 log.cc:824] T 00000000000000000000000000000000 P a5dbd019d6f94f1f9f1fa4849238022f: Log is configured to *not* fsync() on all Append() calls
I20260926 01:14:05.999346 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 21478
I20260926 01:14:05.999469 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790384993574762-392-0/minicluster-data/master-0/wal/instance
I20260926 01:14:05.999644 392 external_mini_cluster.cc:1428] Setting key 546b0e8bcb03bdc3706907cdbb243dcb
I20260926 01:14:05.999923 21546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5dbd019d6f94f1f9f1fa4849238022f: No bootstrap required, opened a new log
I20260926 01:14:06.002043 21546 raft_consensus.cc:348] T 00000000000000000000000000000000 P a5dbd019d6f94f1f9f1fa4849238022f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5dbd019d6f94f1f9f1fa4849238022f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45005 } }
I20260926 01:14:06.002228 21546 raft_consensus.cc:374] T 00000000000000000000000000000000 P a5dbd019d6f94f1f9f1fa4849238022f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:14:06.002298 21546 raft_consensus.cc:729] T 00000000000000000000000000000000 P a5dbd019d6f94f1f9f1fa4849238022f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5dbd019d6f94f1f9f1fa4849238022f, State: Initialized, Role: FOLLOWER
I20260926 01:14:06.002431 21546 consensus_queue.cc:260] T 00000000000000000000000000000000 P a5dbd019d6f94f1f9f1fa4849238022f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5dbd019d6f94f1f9f1fa4849238022f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45005 } }
I20260926 01:14:06.002518 21546 raft_consensus.cc:388] T 00000000000000000000000000000000 P a5dbd019d6f94f1f9f1fa4849238022f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:14:06.002549 21546 raft_consensus.cc:482] T 00000000000000000000000000000000 P a5dbd019d6f94f1f9f1fa4849238022f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:14:06.002548 21550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:14:06.000160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.100:47099 (local address 127.0.0.1:45005)
0926 01:14:06.000331 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 01:14:06.000341 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 01:14:06.000366 (+ 25us) server_negotiation.cc:407] Connection header received
0926 01:14:06.001119 (+ 753us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:14:06.001127 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:14:06.001155 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:14:06.001277 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:14:06.001916 (+ 639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:14:06.001919 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:14:06.001919 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:14:06.001923 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:14:06.001960 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:14:06.001970 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:14:06.002028 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 01:14:06.002110 (+ 82us) server_negotiation.cc:299] Negotiation successful
0926 01:14:06.002150 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":86,"threads_started":1}
I20260926 01:14:06.002619 21546 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a5dbd019d6f94f1f9f1fa4849238022f [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:14:06.003320 21546 raft_consensus.cc:504] T 00000000000000000000000000000000 P a5dbd019d6f94f1f9f1fa4849238022f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5dbd019d6f94f1f9f1fa4849238022f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45005 } }
I20260926 01:14:06.003430 21546 leader_election.cc:302] T 00000000000000000000000000000000 P a5dbd019d6f94f1f9f1fa4849238022f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a5dbd019d6f94f1f9f1fa4849238022f; no voters:
I20260926 01:14:06.003511 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 01:14:06.003597 21546 leader_election.cc:288] T 00000000000000000000000000000000 P a5dbd019d6f94f1f9f1fa4849238022f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:14:06.003741 21552 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a5dbd019d6f94f1f9f1fa4849238022f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:14:06.003849 21546 sys_catalog.cc:564] T 00000000000000000000000000000000 P a5dbd019d6f94f1f9f1fa4849238022f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:14:06.003866 21552 raft_consensus.cc:686] T 00000000000000000000000000000000 P a5dbd019d6f94f1f9f1fa4849238022f [term 1 LEADER]: Becoming Leader. State: Replica: a5dbd019d6f94f1f9f1fa4849238022f, State: Running, Role: LEADER
I20260926 01:14:06.004068 21552 consensus_queue.cc:237] T 00000000000000000000000000000000 P a5dbd019d6f94f1f9f1fa4849238022f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5dbd019d6f94f1f9f1fa4849238022f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45005 } }
I20260926 01:14:06.004447 21553 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5dbd019d6f94f1f9f1fa4849238022f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a5dbd019d6f94f1f9f1fa4849238022f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5dbd019d6f94f1f9f1fa4849238022f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45005 } } }
I20260926 01:14:06.004585 21553 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5dbd019d6f94f1f9f1fa4849238022f [sys.catalog]: This master's current role is: LEADER
I20260926 01:14:06.004765 21553 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5dbd019d6f94f1f9f1fa4849238022f [sys.catalog]: SysCatalogTable state changed. Reason: New leader a5dbd019d6f94f1f9f1fa4849238022f. Latest consensus state: current_term: 1 leader_uuid: "a5dbd019d6f94f1f9f1fa4849238022f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5dbd019d6f94f1f9f1fa4849238022f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45005 } } }
I20260926 01:14:06.004844 21564 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:14:06.004869 21553 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5dbd019d6f94f1f9f1fa4849238022f [sys.catalog]: This master's current role is: LEADER
I20260926 01:14:06.005460 21564 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:14:06.006407 21550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:14:06.005082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.100:39959 (local address 127.0.0.1:45005)
0926 01:14:06.005116 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0926 01:14:06.005118 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:14:06.005207 (+ 89us) server_negotiation.cc:407] Connection header received
0926 01:14:06.005312 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:14:06.005312 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:14:06.005318 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:14:06.005364 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:14:06.006153 (+ 789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:14:06.006154 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:14:06.006154 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:14:06.006158 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:14:06.006173 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:14:06.006178 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:14:06.006230 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 01:14:06.006335 (+ 105us) server_negotiation.cc:299] Negotiation successful
0926 01:14:06.006350 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260926 01:14:06.007177 21564 catalog_manager.cc:1347] Generated new cluster ID: e06e024cc17d4aa3a8a74c4bd2bc9445
I20260926 01:14:06.007229 21564 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 01:14:06.017170 21564 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:14:06.017357 21564 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:14:06.028733 21564 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a5dbd019d6f94f1f9f1fa4849238022f: Generated new TSK 0
I20260926 01:14:06.028936 21564 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 01:14:06.031860 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 21478
2026-09-26T01:14:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1216 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790384993574762-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 01:14:06 dist-test-slave-7b22 krb5kdc[21577](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 01:14:06 dist-test-slave-7b22 krb5kdc[21577](info): set up 2 sockets
Sep 26 01:14:06 dist-test-slave-7b22 krb5kdc[21577](info): commencing operation
krb5kdc: starting...
W20260926 01:14:06.981698 392 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790384993574762-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790384993574762-392-0/krb5kdc/test-user.keytab.
Sep 26 01:14:07 dist-test-slave-7b22 krb5kdc[21577](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385247, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T01:14:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:14:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790384993574762-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790384993574762-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 01:14:07.096689 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790384993574762-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790384993574762-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790384993574762-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42367
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790384993574762-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42367
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:14:07.175043 21590 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:14:07.175258 21590 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:14:07.175289 21590 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:14:07.176561 21590 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:14:07.176610 21590 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:14:07.176623 21590 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:14:07.176635 21590 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:14:07.176645 21590 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:14:07.178174 21590 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790384993574762-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790384993574762-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42367
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790384993574762-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42367
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21590
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790384993574762-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:14:07.178412 21590 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:14:07.178620 21590 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:14:07.183588 21595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:14:07.183588 21598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:14:07.183622 21596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:14:07.183611 21590 server_base.cc:1034] running on GCE node
I20260926 01:14:07.184134 21590 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:14:07.184405 21590 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:14:07.185549 21590 hybrid_clock.cc:648] HybridClock initialized: now 1790385247185535 us; error 30 us; skew 500 ppm
Sep 26 01:14:07 dist-test-slave-7b22 krb5kdc[21577](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385247, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:14:07.188086 21590 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 01:14:07.188524 21590 webserver.cc:460] Webserver started at http://127.0.0.1:43463/ using document root <none> and password file <none>
I20260926 01:14:07.188763 21590 fs_manager.cc:359] Metadata directory not provided
I20260926 01:14:07.188814 21590 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:14:07.188930 21590 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:14:07.190034 21590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790384993574762-392-0/minicluster-data/master-0/data/instance:
uuid: "f2b65eb7a786414baf6c2e9f791d9643"
format_stamp: "Formatted at 2026-09-26 01:14:07 on dist-test-slave-7b22"
server_key: "52ee7584c3d1de8515e2165a8bc0d913"
server_key_iv: "84e681b57e3b1ffc5282cc00c1770829"
server_key_version: "encryptionkey@0"
I20260926 01:14:07.190346 21590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790384993574762-392-0/minicluster-data/master-0/wal/instance:
uuid: "f2b65eb7a786414baf6c2e9f791d9643"
format_stamp: "Formatted at 2026-09-26 01:14:07 on dist-test-slave-7b22"
server_key: "52ee7584c3d1de8515e2165a8bc0d913"
server_key_iv: "84e681b57e3b1ffc5282cc00c1770829"
server_key_version: "encryptionkey@0"
I20260926 01:14:07.191699 21590 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 01:14:07.192482 21605 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:14:07.192672 21590 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:14:07.192762 21590 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790384993574762-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790384993574762-392-0/minicluster-data/master-0/wal
uuid: "f2b65eb7a786414baf6c2e9f791d9643"
format_stamp: "Formatted at 2026-09-26 01:14:07 on dist-test-slave-7b22"
server_key: "52ee7584c3d1de8515e2165a8bc0d913"
server_key_iv: "84e681b57e3b1ffc5282cc00c1770829"
server_key_version: "encryptionkey@0"
I20260926 01:14:07.192829 21590 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790384993574762-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790384993574762-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790384993574762-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:14:07.225337 21590 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:14:07.227300 21590 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:14:07.227414 21590 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:14:07.231129 21590 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42367
I20260926 01:14:07.231164 21657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42367 every 8 connection(s)
I20260926 01:14:07.231516 21590 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
I20260926 01:14:07.232414 21658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:14:07.234767 21658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2b65eb7a786414baf6c2e9f791d9643: Bootstrap starting.
I20260926 01:14:07.235318 21658 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f2b65eb7a786414baf6c2e9f791d9643: Neither blocks nor log segments found. Creating new log.
I20260926 01:14:07.235584 21658 log.cc:824] T 00000000000000000000000000000000 P f2b65eb7a786414baf6c2e9f791d9643: Log is configured to *not* fsync() on all Append() calls
I20260926 01:14:07.236274 21658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2b65eb7a786414baf6c2e9f791d9643: No bootstrap required, opened a new log
I20260926 01:14:07.237896 21658 raft_consensus.cc:348] T 00000000000000000000000000000000 P f2b65eb7a786414baf6c2e9f791d9643 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2b65eb7a786414baf6c2e9f791d9643" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42367 } }
I20260926 01:14:07.238046 21658 raft_consensus.cc:374] T 00000000000000000000000000000000 P f2b65eb7a786414baf6c2e9f791d9643 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:14:07.238068 21658 raft_consensus.cc:729] T 00000000000000000000000000000000 P f2b65eb7a786414baf6c2e9f791d9643 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2b65eb7a786414baf6c2e9f791d9643, State: Initialized, Role: FOLLOWER
I20260926 01:14:07.238207 21658 consensus_queue.cc:260] T 00000000000000000000000000000000 P f2b65eb7a786414baf6c2e9f791d9643 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2b65eb7a786414baf6c2e9f791d9643" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42367 } }
I20260926 01:14:07.238306 21658 raft_consensus.cc:388] T 00000000000000000000000000000000 P f2b65eb7a786414baf6c2e9f791d9643 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:14:07.238341 21658 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2b65eb7a786414baf6c2e9f791d9643 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:14:07.238399 21658 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f2b65eb7a786414baf6c2e9f791d9643 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:14:07.238934 21658 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2b65eb7a786414baf6c2e9f791d9643 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2b65eb7a786414baf6c2e9f791d9643" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42367 } }
I20260926 01:14:07.239027 21658 leader_election.cc:302] T 00000000000000000000000000000000 P f2b65eb7a786414baf6c2e9f791d9643 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f2b65eb7a786414baf6c2e9f791d9643; no voters:
I20260926 01:14:07.239188 21658 leader_election.cc:288] T 00000000000000000000000000000000 P f2b65eb7a786414baf6c2e9f791d9643 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:14:07.239230 21661 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f2b65eb7a786414baf6c2e9f791d9643 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:14:07.239387 21661 raft_consensus.cc:686] T 00000000000000000000000000000000 P f2b65eb7a786414baf6c2e9f791d9643 [term 1 LEADER]: Becoming Leader. State: Replica: f2b65eb7a786414baf6c2e9f791d9643, State: Running, Role: LEADER
I20260926 01:14:07.239455 21658 sys_catalog.cc:564] T 00000000000000000000000000000000 P f2b65eb7a786414baf6c2e9f791d9643 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:14:07.239481 21661 consensus_queue.cc:237] T 00000000000000000000000000000000 P f2b65eb7a786414baf6c2e9f791d9643 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2b65eb7a786414baf6c2e9f791d9643" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42367 } }
I20260926 01:14:07.239828 21663 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2b65eb7a786414baf6c2e9f791d9643 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f2b65eb7a786414baf6c2e9f791d9643. Latest consensus state: current_term: 1 leader_uuid: "f2b65eb7a786414baf6c2e9f791d9643" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2b65eb7a786414baf6c2e9f791d9643" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42367 } } }
I20260926 01:14:07.239912 21663 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2b65eb7a786414baf6c2e9f791d9643 [sys.catalog]: This master's current role is: LEADER
I20260926 01:14:07.239961 21662 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2b65eb7a786414baf6c2e9f791d9643 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f2b65eb7a786414baf6c2e9f791d9643" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2b65eb7a786414baf6c2e9f791d9643" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42367 } } }
I20260926 01:14:07.240008 21662 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2b65eb7a786414baf6c2e9f791d9643 [sys.catalog]: This master's current role is: LEADER
I20260926 01:14:07.240121 21668 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:14:07.240631 21668 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:14:07.241675 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 21590
I20260926 01:14:07.241789 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790384993574762-392-0/minicluster-data/master-0/wal/instance
I20260926 01:14:07.241986 392 external_mini_cluster.cc:1428] Setting key 78c45faee9fbf4af3fc83c70a1eaf339
I20260926 01:14:07.242249 21668 catalog_manager.cc:1347] Generated new cluster ID: 9d9ebee498034297be542e08c8224c94
I20260926 01:14:07.242298 21668 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 01:14:07 dist-test-slave-7b22 krb5kdc[21577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385247, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 01:14:07.253878 21681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:14:07.242630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.100:42887 (local address 127.0.0.1:42367)
0926 01:14:07.242793 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0926 01:14:07.242802 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 01:14:07.242834 (+ 32us) server_negotiation.cc:407] Connection header received
0926 01:14:07.242951 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:14:07.242957 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:14:07.243013 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:14:07.243141 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:14:07.243895 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:14:07.245306 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:14:07.246424 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:14:07.247046 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:14:07.247142 (+ 96us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:14:07.250031 (+ 2889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:14:07.250036 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:14:07.250036 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:14:07.250050 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:14:07.251975 (+ 1925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:14:07.252423 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:14:07.252427 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:14:07.252428 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:14:07.252551 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:14:07.252750 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:14:07.252751 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:14:07.252751 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:14:07.253093 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:14:07.253258 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0926 01:14:07.253348 (+ 90us) server_negotiation.cc:299] Negotiation successful
0926 01:14:07.253580 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":78,"threads_started":1}
I20260926 01:14:07.254853 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 01:14:07.258098 21668 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:14:07.258796 21668 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:14:07.263453 21681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:14:07.256345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.100:54409 (local address 127.0.0.1:42367)
0926 01:14:07.256387 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 01:14:07.256389 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:14:07.256592 (+ 203us) server_negotiation.cc:407] Connection header received
0926 01:14:07.256639 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:14:07.256640 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:14:07.256651 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:14:07.256707 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:14:07.257371 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:14:07.258007 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:14:07.258982 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:14:07.259286 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:14:07.259321 (+ 35us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:14:07.260751 (+ 1430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:14:07.260752 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:14:07.260753 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:14:07.260759 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:14:07.262212 (+ 1453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:14:07.262613 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:14:07.262614 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:14:07.262614 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:14:07.262662 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:14:07.262879 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:14:07.262880 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:14:07.262880 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:14:07.262999 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:14:07.263086 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 01:14:07.263291 (+ 205us) server_negotiation.cc:299] Negotiation successful
0926 01:14:07.263352 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 01:14:07.265470 21668 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f2b65eb7a786414baf6c2e9f791d9643: Generated new TSK 0
I20260926 01:14:07.265638 21668 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 01:14:07.270830 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 21590
W20260926 01:14:07.274474 21575 connection.cc:535] client connection to 127.0.0.1:42367 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42367): Connection reset by peer (error 104)
2026-09-26T01:14:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1238 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790384993574762-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 01:14:07 dist-test-slave-7b22 krb5kdc[21689](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 01:14:07 dist-test-slave-7b22 krb5kdc[21689](info): set up 2 sockets
Sep 26 01:14:07 dist-test-slave-7b22 krb5kdc[21689](info): commencing operation
krb5kdc: starting...
W20260926 01:14:08.233724 392 mini_kdc.cc:121] Time spent starting KDC: real 0.946s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790384993574762-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790384993574762-392-0/krb5kdc/test-user.keytab.
Sep 26 01:14:08 dist-test-slave-7b22 krb5kdc[21689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385248, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T01:14:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:14:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790384993574762-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790384993574762-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 01:14:08.352779 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790384993574762-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790384993574762-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790384993574762-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40637
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790384993574762-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40637
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:14:08.432574 21702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:14:08.432780 21702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:14:08.432802 21702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:14:08.433954 21702 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:14:08.433991 21702 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:14:08.434003 21702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:14:08.434015 21702 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:14:08.434026 21702 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:14:08.435514 21702 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790384993574762-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790384993574762-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40637
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790384993574762-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40637
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21702
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790384993574762-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:14:08.435737 21702 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:14:08.435990 21702 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:14:08.440929 21707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:14:08.441058 21702 server_base.cc:1034] running on GCE node
W20260926 01:14:08.440935 21710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:14:08.440949 21708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:14:08.441407 21702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:14:08.441664 21702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:14:08.442803 21702 hybrid_clock.cc:648] HybridClock initialized: now 1790385248442773 us; error 45 us; skew 500 ppm
Sep 26 01:14:08 dist-test-slave-7b22 krb5kdc[21689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385248, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:14:08.445151 21702 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 01:14:08.445564 21702 webserver.cc:460] Webserver started at http://127.0.0.1:44275/ using document root <none> and password file <none>
I20260926 01:14:08.445761 21702 fs_manager.cc:359] Metadata directory not provided
I20260926 01:14:08.445802 21702 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:14:08.445945 21702 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:14:08.446975 21702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790384993574762-392-0/minicluster-data/master-0/data/instance:
uuid: "95da5874399042bb884808cf0e91eac4"
format_stamp: "Formatted at 2026-09-26 01:14:08 on dist-test-slave-7b22"
server_key: "db431625d23f130fe46d66bcd465beba"
server_key_iv: "8e1b19d0db0074bb461e054b2fee31f4"
server_key_version: "encryptionkey@0"
I20260926 01:14:08.447327 21702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790384993574762-392-0/minicluster-data/master-0/wal/instance:
uuid: "95da5874399042bb884808cf0e91eac4"
format_stamp: "Formatted at 2026-09-26 01:14:08 on dist-test-slave-7b22"
server_key: "db431625d23f130fe46d66bcd465beba"
server_key_iv: "8e1b19d0db0074bb461e054b2fee31f4"
server_key_version: "encryptionkey@0"
I20260926 01:14:08.448573 21702 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 01:14:08.449252 21717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:14:08.449421 21702 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 01:14:08.449498 21702 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790384993574762-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790384993574762-392-0/minicluster-data/master-0/wal
uuid: "95da5874399042bb884808cf0e91eac4"
format_stamp: "Formatted at 2026-09-26 01:14:08 on dist-test-slave-7b22"
server_key: "db431625d23f130fe46d66bcd465beba"
server_key_iv: "8e1b19d0db0074bb461e054b2fee31f4"
server_key_version: "encryptionkey@0"
I20260926 01:14:08.449555 21702 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790384993574762-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790384993574762-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790384993574762-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:14:08.471195 21702 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:14:08.473282 21702 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:14:08.473418 21702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:14:08.477090 21702 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40637
I20260926 01:14:08.477128 21769 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40637 every 8 connection(s)
I20260926 01:14:08.477473 21702 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
I20260926 01:14:08.478466 21770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:14:08.480661 21770 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95da5874399042bb884808cf0e91eac4: Bootstrap starting.
I20260926 01:14:08.481208 21770 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 95da5874399042bb884808cf0e91eac4: Neither blocks nor log segments found. Creating new log.
I20260926 01:14:08.481464 21770 log.cc:824] T 00000000000000000000000000000000 P 95da5874399042bb884808cf0e91eac4: Log is configured to *not* fsync() on all Append() calls
I20260926 01:14:08.482157 21770 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95da5874399042bb884808cf0e91eac4: No bootstrap required, opened a new log
I20260926 01:14:08.483764 21770 raft_consensus.cc:348] T 00000000000000000000000000000000 P 95da5874399042bb884808cf0e91eac4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95da5874399042bb884808cf0e91eac4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40637 } }
I20260926 01:14:08.483909 21770 raft_consensus.cc:374] T 00000000000000000000000000000000 P 95da5874399042bb884808cf0e91eac4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:14:08.483942 21770 raft_consensus.cc:729] T 00000000000000000000000000000000 P 95da5874399042bb884808cf0e91eac4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95da5874399042bb884808cf0e91eac4, State: Initialized, Role: FOLLOWER
I20260926 01:14:08.484081 21770 consensus_queue.cc:260] T 00000000000000000000000000000000 P 95da5874399042bb884808cf0e91eac4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95da5874399042bb884808cf0e91eac4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40637 } }
I20260926 01:14:08.484140 21770 raft_consensus.cc:388] T 00000000000000000000000000000000 P 95da5874399042bb884808cf0e91eac4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:14:08.484193 21770 raft_consensus.cc:482] T 00000000000000000000000000000000 P 95da5874399042bb884808cf0e91eac4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:14:08.484243 21770 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 95da5874399042bb884808cf0e91eac4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:14:08.484762 21770 raft_consensus.cc:504] T 00000000000000000000000000000000 P 95da5874399042bb884808cf0e91eac4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95da5874399042bb884808cf0e91eac4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40637 } }
I20260926 01:14:08.484858 21770 leader_election.cc:302] T 00000000000000000000000000000000 P 95da5874399042bb884808cf0e91eac4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 95da5874399042bb884808cf0e91eac4; no voters:
I20260926 01:14:08.485030 21770 leader_election.cc:288] T 00000000000000000000000000000000 P 95da5874399042bb884808cf0e91eac4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:14:08.485073 21773 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 95da5874399042bb884808cf0e91eac4 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:14:08.485235 21773 raft_consensus.cc:686] T 00000000000000000000000000000000 P 95da5874399042bb884808cf0e91eac4 [term 1 LEADER]: Becoming Leader. State: Replica: 95da5874399042bb884808cf0e91eac4, State: Running, Role: LEADER
I20260926 01:14:08.485275 21770 sys_catalog.cc:564] T 00000000000000000000000000000000 P 95da5874399042bb884808cf0e91eac4 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:14:08.485327 21773 consensus_queue.cc:237] T 00000000000000000000000000000000 P 95da5874399042bb884808cf0e91eac4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95da5874399042bb884808cf0e91eac4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40637 } }
I20260926 01:14:08.485688 21775 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95da5874399042bb884808cf0e91eac4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 95da5874399042bb884808cf0e91eac4. Latest consensus state: current_term: 1 leader_uuid: "95da5874399042bb884808cf0e91eac4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95da5874399042bb884808cf0e91eac4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40637 } } }
I20260926 01:14:08.485845 21774 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95da5874399042bb884808cf0e91eac4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "95da5874399042bb884808cf0e91eac4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95da5874399042bb884808cf0e91eac4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40637 } } }
I20260926 01:14:08.485862 21775 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95da5874399042bb884808cf0e91eac4 [sys.catalog]: This master's current role is: LEADER
I20260926 01:14:08.485966 21774 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95da5874399042bb884808cf0e91eac4 [sys.catalog]: This master's current role is: LEADER
I20260926 01:14:08.486287 21782 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:14:08.486753 21782 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:14:08.487634 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 21702
I20260926 01:14:08.487759 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790384993574762-392-0/minicluster-data/master-0/wal/instance
I20260926 01:14:08.487932 392 external_mini_cluster.cc:1428] Setting key f1693c0ff8153925ce474c96fe4f9490
I20260926 01:14:08.488488 21782 catalog_manager.cc:1347] Generated new cluster ID: 766ddcbaada24610b0b306cd85ff6928
I20260926 01:14:08.488541 21782 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 01:14:08.499439 21793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:14:08.488405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.100:40521 (local address 127.0.0.1:40637)
0926 01:14:08.488767 (+ 362us) server_negotiation.cc:206] Beginning negotiation
0926 01:14:08.488776 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 01:14:08.488808 (+ 32us) server_negotiation.cc:407] Connection header received
0926 01:14:08.488924 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:14:08.488931 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:14:08.488988 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:14:08.489121 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:14:08.489360 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:14:08.490865 (+ 1505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:14:08.491914 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:14:08.492576 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:14:08.492687 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:14:08.498198 (+ 5511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:14:08.498235 (+ 37us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:14:08.498235 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:14:08.498599 (+ 364us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:14:08.498665 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:14:08.498684 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:14:08.499090 (+ 406us) server_negotiation.cc:1035] Waiting for connection context
0926 01:14:08.499139 (+ 49us) server_negotiation.cc:299] Negotiation successful
0926 01:14:08.499171 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":75,"threads_started":1}
I20260926 01:14:08.502208 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 01:14:08.507705 21793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:14:08.504191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.100:43777 (local address 127.0.0.1:40637)
0926 01:14:08.504240 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 01:14:08.504242 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:14:08.504343 (+ 101us) server_negotiation.cc:407] Connection header received
0926 01:14:08.504387 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:14:08.504388 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:14:08.504399 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:14:08.504464 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:14:08.504596 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:14:08.505375 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:14:08.506386 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:14:08.506720 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:14:08.506790 (+ 70us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:14:08.507451 (+ 661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:14:08.507452 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:14:08.507452 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:14:08.507458 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:14:08.507475 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:14:08.507480 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:14:08.507539 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 01:14:08.507600 (+ 61us) server_negotiation.cc:299] Negotiation successful
0926 01:14:08.507615 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 01:14:08.518823 21782 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:14:08.519551 21782 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:14:08.524757 21782 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 95da5874399042bb884808cf0e91eac4: Generated new TSK 0
I20260926 01:14:08.524963 21782 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 01:14:08.529011 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 21702
W20260926 01:14:08.532946 21687 connection.cc:535] client connection to 127.0.0.1:40637 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40637): Connection reset by peer (error 104)
2026-09-26T01:14:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1258 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790384993574762-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 01:14:08 dist-test-slave-7b22 krb5kdc[21801](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 01:14:08 dist-test-slave-7b22 krb5kdc[21801](info): set up 2 sockets
Sep 26 01:14:08 dist-test-slave-7b22 krb5kdc[21801](info): commencing operation
krb5kdc: starting...
W20260926 01:14:09.477682 392 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790384993574762-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790384993574762-392-0/krb5kdc/test-user.keytab.
Sep 26 01:14:09 dist-test-slave-7b22 krb5kdc[21801](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385249, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T01:14:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:14:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790384993574762-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790384993574762-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 01:14:09.596975 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790384993574762-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790384993574762-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790384993574762-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35635
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790384993574762-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35635
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:14:09.676178 21814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:14:09.676386 21814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:14:09.676409 21814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:14:09.677568 21814 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:14:09.677615 21814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:14:09.677628 21814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:14:09.677639 21814 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:14:09.677650 21814 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:14:09.679518 21814 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790384993574762-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790384993574762-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35635
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790384993574762-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35635
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21814
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790384993574762-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:14:09.679759 21814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:14:09.679981 21814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:14:09.685104 21822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:14:09.685097 21820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:14:09.685097 21819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:14:09.685449 21814 server_base.cc:1034] running on GCE node
I20260926 01:14:09.685689 21814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:14:09.685987 21814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:14:09.687150 21814 hybrid_clock.cc:648] HybridClock initialized: now 1790385249687136 us; error 30 us; skew 500 ppm
Sep 26 01:14:09 dist-test-slave-7b22 krb5kdc[21801](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385249, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:14:09.689633 21814 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 01:14:09.690091 21814 webserver.cc:460] Webserver started at http://127.0.0.1:43903/ using document root <none> and password file <none>
I20260926 01:14:09.690296 21814 fs_manager.cc:359] Metadata directory not provided
I20260926 01:14:09.690344 21814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:14:09.690474 21814 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:14:09.691525 21814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790384993574762-392-0/minicluster-data/master-0/data/instance:
uuid: "35d036e4ee87463fa03ae66cd2442849"
format_stamp: "Formatted at 2026-09-26 01:14:09 on dist-test-slave-7b22"
server_key: "da699bd6ed9209687264ef265ed50dab"
server_key_iv: "fe970d0e15be6355d993c8e8219c9868"
server_key_version: "encryptionkey@0"
I20260926 01:14:09.691838 21814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790384993574762-392-0/minicluster-data/master-0/wal/instance:
uuid: "35d036e4ee87463fa03ae66cd2442849"
format_stamp: "Formatted at 2026-09-26 01:14:09 on dist-test-slave-7b22"
server_key: "da699bd6ed9209687264ef265ed50dab"
server_key_iv: "fe970d0e15be6355d993c8e8219c9868"
server_key_version: "encryptionkey@0"
I20260926 01:14:09.693084 21814 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 01:14:09.693852 21829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:14:09.694007 21814 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:14:09.694089 21814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790384993574762-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790384993574762-392-0/minicluster-data/master-0/wal
uuid: "35d036e4ee87463fa03ae66cd2442849"
format_stamp: "Formatted at 2026-09-26 01:14:09 on dist-test-slave-7b22"
server_key: "da699bd6ed9209687264ef265ed50dab"
server_key_iv: "fe970d0e15be6355d993c8e8219c9868"
server_key_version: "encryptionkey@0"
I20260926 01:14:09.694156 21814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790384993574762-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790384993574762-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790384993574762-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:14:09.697275 21814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:14:09.699363 21814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:14:09.699491 21814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:14:09.703244 21814 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35635
I20260926 01:14:09.703286 21881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35635 every 8 connection(s)
I20260926 01:14:09.703590 21814 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
I20260926 01:14:09.704567 21882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:14:09.707051 21882 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35d036e4ee87463fa03ae66cd2442849: Bootstrap starting.
I20260926 01:14:09.707587 21882 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 35d036e4ee87463fa03ae66cd2442849: Neither blocks nor log segments found. Creating new log.
I20260926 01:14:09.707821 21882 log.cc:824] T 00000000000000000000000000000000 P 35d036e4ee87463fa03ae66cd2442849: Log is configured to *not* fsync() on all Append() calls
I20260926 01:14:09.708490 21882 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35d036e4ee87463fa03ae66cd2442849: No bootstrap required, opened a new log
I20260926 01:14:09.710143 21882 raft_consensus.cc:348] T 00000000000000000000000000000000 P 35d036e4ee87463fa03ae66cd2442849 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35d036e4ee87463fa03ae66cd2442849" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35635 } }
I20260926 01:14:09.710299 21882 raft_consensus.cc:374] T 00000000000000000000000000000000 P 35d036e4ee87463fa03ae66cd2442849 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:14:09.710321 21882 raft_consensus.cc:729] T 00000000000000000000000000000000 P 35d036e4ee87463fa03ae66cd2442849 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35d036e4ee87463fa03ae66cd2442849, State: Initialized, Role: FOLLOWER
I20260926 01:14:09.710434 21882 consensus_queue.cc:260] T 00000000000000000000000000000000 P 35d036e4ee87463fa03ae66cd2442849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35d036e4ee87463fa03ae66cd2442849" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35635 } }
I20260926 01:14:09.710525 21882 raft_consensus.cc:388] T 00000000000000000000000000000000 P 35d036e4ee87463fa03ae66cd2442849 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:14:09.710553 21882 raft_consensus.cc:482] T 00000000000000000000000000000000 P 35d036e4ee87463fa03ae66cd2442849 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:14:09.710584 21882 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 35d036e4ee87463fa03ae66cd2442849 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:14:09.711086 21882 raft_consensus.cc:504] T 00000000000000000000000000000000 P 35d036e4ee87463fa03ae66cd2442849 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35d036e4ee87463fa03ae66cd2442849" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35635 } }
I20260926 01:14:09.711172 21882 leader_election.cc:302] T 00000000000000000000000000000000 P 35d036e4ee87463fa03ae66cd2442849 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 35d036e4ee87463fa03ae66cd2442849; no voters:
I20260926 01:14:09.711309 21882 leader_election.cc:288] T 00000000000000000000000000000000 P 35d036e4ee87463fa03ae66cd2442849 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:14:09.711369 21885 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 35d036e4ee87463fa03ae66cd2442849 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:14:09.711577 21882 sys_catalog.cc:564] T 00000000000000000000000000000000 P 35d036e4ee87463fa03ae66cd2442849 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:14:09.711618 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 21814
I20260926 01:14:09.711731 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790384993574762-392-0/minicluster-data/master-0/wal/instance
I20260926 01:14:09.711755 21885 raft_consensus.cc:686] T 00000000000000000000000000000000 P 35d036e4ee87463fa03ae66cd2442849 [term 1 LEADER]: Becoming Leader. State: Replica: 35d036e4ee87463fa03ae66cd2442849, State: Running, Role: LEADER
I20260926 01:14:09.711900 392 external_mini_cluster.cc:1428] Setting key f043b1fcc7b82342584ec50c74ff2781
I20260926 01:14:09.711908 21885 consensus_queue.cc:237] T 00000000000000000000000000000000 P 35d036e4ee87463fa03ae66cd2442849 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35d036e4ee87463fa03ae66cd2442849" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35635 } }
I20260926 01:14:09.712266 21886 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35d036e4ee87463fa03ae66cd2442849 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "35d036e4ee87463fa03ae66cd2442849" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35d036e4ee87463fa03ae66cd2442849" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35635 } } }
I20260926 01:14:09.712348 21886 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35d036e4ee87463fa03ae66cd2442849 [sys.catalog]: This master's current role is: LEADER
I20260926 01:14:09.712482 21887 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35d036e4ee87463fa03ae66cd2442849 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 35d036e4ee87463fa03ae66cd2442849. Latest consensus state: current_term: 1 leader_uuid: "35d036e4ee87463fa03ae66cd2442849" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35d036e4ee87463fa03ae66cd2442849" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35635 } } }
I20260926 01:14:09.712543 21887 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35d036e4ee87463fa03ae66cd2442849 [sys.catalog]: This master's current role is: LEADER
I20260926 01:14:09.712987 21892 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:14:09.713513 21892 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:14:09.715174 21892 catalog_manager.cc:1347] Generated new cluster ID: c70b7e30a597418faa6d384e04094853
I20260926 01:14:09.715227 21892 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 01:14:09.715420 21891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:14:09.712344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.100:53573 (local address 127.0.0.1:35635)
0926 01:14:09.713523 (+ 1179us) server_negotiation.cc:206] Beginning negotiation
0926 01:14:09.713533 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 01:14:09.713570 (+ 37us) server_negotiation.cc:407] Connection header received
0926 01:14:09.713691 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:14:09.713699 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:14:09.713730 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:14:09.713931 (+ 201us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:14:09.714882 (+ 951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:14:09.714885 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:14:09.714886 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:14:09.714890 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:14:09.714934 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:14:09.714949 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:14:09.715015 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 01:14:09.715090 (+ 75us) server_negotiation.cc:299] Negotiation successful
0926 01:14:09.715225 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1126,"thread_start_us":74,"threads_started":1}
I20260926 01:14:09.716312 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 01:14:09.719085 21891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:14:09.717746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.100:50385 (local address 127.0.0.1:35635)
0926 01:14:09.717789 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 01:14:09.717791 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:14:09.718151 (+ 360us) server_negotiation.cc:407] Connection header received
0926 01:14:09.718222 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:14:09.718223 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:14:09.718229 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:14:09.718283 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:14:09.718872 (+ 589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:14:09.718873 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:14:09.718874 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:14:09.718877 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:14:09.718891 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:14:09.718895 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:14:09.718942 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0926 01:14:09.718987 (+ 45us) server_negotiation.cc:299] Negotiation successful
0926 01:14:09.718999 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 01:14:09.731200 21892 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:14:09.731407 21892 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:14:09.735832 21892 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 35d036e4ee87463fa03ae66cd2442849: Generated new TSK 0
I20260926 01:14:09.736078 21892 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 01:14:09.737661 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 21814
2026-09-26T01:14:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1208 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790384993574762-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 01:14:09 dist-test-slave-7b22 krb5kdc[21913](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 01:14:09 dist-test-slave-7b22 krb5kdc[21913](info): set up 2 sockets
Sep 26 01:14:09 dist-test-slave-7b22 krb5kdc[21913](info): commencing operation
krb5kdc: starting...
W20260926 01:14:10.689743 392 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790384993574762-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790384993574762-392-0/krb5kdc/test-user.keytab.
Sep 26 01:14:10 dist-test-slave-7b22 krb5kdc[21913](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T01:14:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:14:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790384993574762-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790384993574762-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 01:14:10.816948 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790384993574762-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790384993574762-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790384993574762-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33921
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790384993574762-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33921
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:14:10.897687 21926 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:14:10.897945 21926 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:14:10.897970 21926 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:14:10.899132 21926 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:14:10.899169 21926 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:14:10.899183 21926 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:14:10.899194 21926 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:14:10.899204 21926 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:14:10.900641 21926 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790384993574762-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790384993574762-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33921
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790384993574762-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33921
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21926
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790384993574762-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:14:10.900849 21926 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:14:10.901034 21926 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:14:10.905918 21931 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:14:10.905968 21926 server_base.cc:1034] running on GCE node
W20260926 01:14:10.905925 21934 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:14:10.905936 21932 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:14:10.906505 21926 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:14:10.906797 21926 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:14:10.907954 21926 hybrid_clock.cc:648] HybridClock initialized: now 1790385250907937 us; error 34 us; skew 500 ppm
Sep 26 01:14:10 dist-test-slave-7b22 krb5kdc[21913](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385250, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:14:10.910578 21926 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 01:14:10.911053 21926 webserver.cc:460] Webserver started at http://127.0.98.62:36171/ using document root <none> and password file <none>
I20260926 01:14:10.911259 21926 fs_manager.cc:359] Metadata directory not provided
I20260926 01:14:10.911304 21926 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:14:10.911433 21926 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:14:10.912524 21926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790384993574762-392-0/minicluster-data/master-0/data/instance:
uuid: "015822088c4041e18a2119fe87370612"
format_stamp: "Formatted at 2026-09-26 01:14:10 on dist-test-slave-7b22"
server_key: "9cde743965c30751bffad3e58b1870e7"
server_key_iv: "9f52a4e3d742077c26662e108d4abe1c"
server_key_version: "encryptionkey@0"
I20260926 01:14:10.912855 21926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790384993574762-392-0/minicluster-data/master-0/wal/instance:
uuid: "015822088c4041e18a2119fe87370612"
format_stamp: "Formatted at 2026-09-26 01:14:10 on dist-test-slave-7b22"
server_key: "9cde743965c30751bffad3e58b1870e7"
server_key_iv: "9f52a4e3d742077c26662e108d4abe1c"
server_key_version: "encryptionkey@0"
I20260926 01:14:10.914215 21926 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 01:14:10.914923 21941 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:14:10.915091 21926 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 01:14:10.915175 21926 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790384993574762-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790384993574762-392-0/minicluster-data/master-0/wal
uuid: "015822088c4041e18a2119fe87370612"
format_stamp: "Formatted at 2026-09-26 01:14:10 on dist-test-slave-7b22"
server_key: "9cde743965c30751bffad3e58b1870e7"
server_key_iv: "9f52a4e3d742077c26662e108d4abe1c"
server_key_version: "encryptionkey@0"
I20260926 01:14:10.915243 21926 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790384993574762-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790384993574762-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790384993574762-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:14:10.927253 21926 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:14:10.929469 21926 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:14:10.929616 21926 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:14:10.933374 21926 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33921
I20260926 01:14:10.933444 21993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33921 every 8 connection(s)
I20260926 01:14:10.933749 21926 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
I20260926 01:14:10.934648 21994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:14:10.936982 21994 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 015822088c4041e18a2119fe87370612: Bootstrap starting.
I20260926 01:14:10.937530 21994 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 015822088c4041e18a2119fe87370612: Neither blocks nor log segments found. Creating new log.
I20260926 01:14:10.937759 21994 log.cc:824] T 00000000000000000000000000000000 P 015822088c4041e18a2119fe87370612: Log is configured to *not* fsync() on all Append() calls
I20260926 01:14:10.938560 21994 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 015822088c4041e18a2119fe87370612: No bootstrap required, opened a new log
I20260926 01:14:10.940227 21994 raft_consensus.cc:348] T 00000000000000000000000000000000 P 015822088c4041e18a2119fe87370612 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "015822088c4041e18a2119fe87370612" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33921 } }
I20260926 01:14:10.940378 21994 raft_consensus.cc:374] T 00000000000000000000000000000000 P 015822088c4041e18a2119fe87370612 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:14:10.940400 21994 raft_consensus.cc:729] T 00000000000000000000000000000000 P 015822088c4041e18a2119fe87370612 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 015822088c4041e18a2119fe87370612, State: Initialized, Role: FOLLOWER
I20260926 01:14:10.940510 21994 consensus_queue.cc:260] T 00000000000000000000000000000000 P 015822088c4041e18a2119fe87370612 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "015822088c4041e18a2119fe87370612" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33921 } }
I20260926 01:14:10.940565 21994 raft_consensus.cc:388] T 00000000000000000000000000000000 P 015822088c4041e18a2119fe87370612 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:14:10.940594 21994 raft_consensus.cc:482] T 00000000000000000000000000000000 P 015822088c4041e18a2119fe87370612 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:14:10.940634 21994 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 015822088c4041e18a2119fe87370612 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:14:10.941215 21994 raft_consensus.cc:504] T 00000000000000000000000000000000 P 015822088c4041e18a2119fe87370612 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "015822088c4041e18a2119fe87370612" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33921 } }
I20260926 01:14:10.941316 21994 leader_election.cc:302] T 00000000000000000000000000000000 P 015822088c4041e18a2119fe87370612 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 015822088c4041e18a2119fe87370612; no voters:
I20260926 01:14:10.941484 21994 leader_election.cc:288] T 00000000000000000000000000000000 P 015822088c4041e18a2119fe87370612 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:14:10.941552 21997 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 015822088c4041e18a2119fe87370612 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:14:10.941713 21997 raft_consensus.cc:686] T 00000000000000000000000000000000 P 015822088c4041e18a2119fe87370612 [term 1 LEADER]: Becoming Leader. State: Replica: 015822088c4041e18a2119fe87370612, State: Running, Role: LEADER
I20260926 01:14:10.941743 21994 sys_catalog.cc:564] T 00000000000000000000000000000000 P 015822088c4041e18a2119fe87370612 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:14:10.942011 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 21926
I20260926 01:14:10.942101 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790384993574762-392-0/minicluster-data/master-0/wal/instance
I20260926 01:14:10.942271 392 external_mini_cluster.cc:1428] Setting key b6f45e134fe92d7b95d0f9cfa1325acd
I20260926 01:14:10.942472 21997 consensus_queue.cc:237] T 00000000000000000000000000000000 P 015822088c4041e18a2119fe87370612 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "015822088c4041e18a2119fe87370612" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33921 } }
I20260926 01:14:10.943017 21999 sys_catalog.cc:455] T 00000000000000000000000000000000 P 015822088c4041e18a2119fe87370612 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 015822088c4041e18a2119fe87370612. Latest consensus state: current_term: 1 leader_uuid: "015822088c4041e18a2119fe87370612" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "015822088c4041e18a2119fe87370612" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33921 } } }
I20260926 01:14:10.943099 21999 sys_catalog.cc:458] T 00000000000000000000000000000000 P 015822088c4041e18a2119fe87370612 [sys.catalog]: This master's current role is: LEADER
I20260926 01:14:10.943331 21998 sys_catalog.cc:455] T 00000000000000000000000000000000 P 015822088c4041e18a2119fe87370612 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "015822088c4041e18a2119fe87370612" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "015822088c4041e18a2119fe87370612" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33921 } } }
I20260926 01:14:10.943753 21998 sys_catalog.cc:458] T 00000000000000000000000000000000 P 015822088c4041e18a2119fe87370612 [sys.catalog]: This master's current role is: LEADER
W20260926 01:14:10.943670 22012 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 015822088c4041e18a2119fe87370612: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 01:14:10.944211 22012 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 01:14:10.944278 22013 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:14:10.944437 22013 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:14:10.946020 22013 catalog_manager.cc:1347] Generated new cluster ID: e181d537062247beb3acf8a991d4138d
I20260926 01:14:10.946072 22013 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 01:14:10 dist-test-slave-7b22 krb5kdc[21913](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:14:10.956838 22015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:14:10.943456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43165 (local address 127.0.98.62:33921)
0926 01:14:10.943703 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0926 01:14:10.943713 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 01:14:10.943747 (+ 34us) server_negotiation.cc:407] Connection header received
0926 01:14:10.943873 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:14:10.943881 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:14:10.943938 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:14:10.944098 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:14:10.944884 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:14:10.946235 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:14:10.947286 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:14:10.947938 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:14:10.948048 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:14:10.951050 (+ 3002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:14:10.951056 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:14:10.951056 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:14:10.951071 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:14:10.954986 (+ 3915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:14:10.955525 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:14:10.955529 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:14:10.955529 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:14:10.955652 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:14:10.955918 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:14:10.955919 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:14:10.955920 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:14:10.956206 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:14:10.956365 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0926 01:14:10.956496 (+ 131us) server_negotiation.cc:299] Negotiation successful
0926 01:14:10.956598 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":77,"threads_started":1}
I20260926 01:14:10.958027 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 01:14:10.965055 22015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:14:10.959332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50757 (local address 127.0.98.62:33921)
0926 01:14:10.959374 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 01:14:10.959375 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 01:14:10.959452 (+ 77us) server_negotiation.cc:407] Connection header received
0926 01:14:10.959507 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:14:10.959508 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:14:10.959517 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:14:10.959562 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:14:10.960089 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:14:10.960599 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:14:10.961277 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:14:10.961492 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:14:10.961548 (+ 56us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:14:10.962746 (+ 1198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:14:10.962748 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:14:10.962748 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:14:10.962756 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:14:10.964013 (+ 1257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:14:10.964349 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:14:10.964350 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:14:10.964351 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:14:10.964400 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:14:10.964595 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:14:10.964595 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:14:10.964596 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:14:10.964700 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:14:10.964803 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 01:14:10.964908 (+ 105us) server_negotiation.cc:299] Negotiation successful
0926 01:14:10.964962 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 01:14:10.982481 22013 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:14:10.983048 22013 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:14:10.995180 22013 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 015822088c4041e18a2119fe87370612: Generated new TSK 0
I20260926 01:14:10.995424 22013 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 01:14:11.002635 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 21926
W20260926 01:14:11.006541 21911 connection.cc:535] client connection to 127.0.98.62:33921 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:33921): Connection reset by peer (error 104)
2026-09-26T01:14:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1264 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790384993574762-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 01:14:11 dist-test-slave-7b22 krb5kdc[22025](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 01:14:11 dist-test-slave-7b22 krb5kdc[22025](info): set up 2 sockets
Sep 26 01:14:11 dist-test-slave-7b22 krb5kdc[22025](info): commencing operation
krb5kdc: starting...
W20260926 01:14:11.961761 392 mini_kdc.cc:121] Time spent starting KDC: real 0.942s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790384993574762-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790384993574762-392-0/krb5kdc/test-user.keytab.
Sep 26 01:14:12 dist-test-slave-7b22 krb5kdc[22025](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385252, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T01:14:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:14:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790384993574762-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790384993574762-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 01:14:12.080942 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790384993574762-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790384993574762-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790384993574762-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40939
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790384993574762-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40939
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:14:12.163208 22038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:14:12.163441 22038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:14:12.163475 22038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:14:12.164697 22038 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:14:12.164745 22038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:14:12.164764 22038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:14:12.164780 22038 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:14:12.164795 22038 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:14:12.166424 22038 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790384993574762-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790384993574762-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40939
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790384993574762-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40939
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22038
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790384993574762-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:14:12.166692 22038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:14:12.166925 22038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:14:12.171936 22043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:14:12.171918 22046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:14:12.171921 22044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:14:12.171976 22038 server_base.cc:1034] running on GCE node
I20260926 01:14:12.172389 22038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:14:12.172653 22038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:14:12.173832 22038 hybrid_clock.cc:648] HybridClock initialized: now 1790385252173789 us; error 31 us; skew 500 ppm
Sep 26 01:14:12 dist-test-slave-7b22 krb5kdc[22025](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385252, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:14:12.176385 22038 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 01:14:12.176844 22038 webserver.cc:460] Webserver started at http://127.0.98.62:35857/ using document root <none> and password file <none>
I20260926 01:14:12.177042 22038 fs_manager.cc:359] Metadata directory not provided
I20260926 01:14:12.177089 22038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:14:12.177240 22038 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:14:12.178375 22038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790384993574762-392-0/minicluster-data/master-0/data/instance:
uuid: "635a2b70804f4c29bb3f0903e27dd56a"
format_stamp: "Formatted at 2026-09-26 01:14:12 on dist-test-slave-7b22"
server_key: "0e2f00f7cbe8391bc2adbced553be94b"
server_key_iv: "4610c461dfede924a0c5da44880cc5d2"
server_key_version: "encryptionkey@0"
I20260926 01:14:12.178701 22038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790384993574762-392-0/minicluster-data/master-0/wal/instance:
uuid: "635a2b70804f4c29bb3f0903e27dd56a"
format_stamp: "Formatted at 2026-09-26 01:14:12 on dist-test-slave-7b22"
server_key: "0e2f00f7cbe8391bc2adbced553be94b"
server_key_iv: "4610c461dfede924a0c5da44880cc5d2"
server_key_version: "encryptionkey@0"
I20260926 01:14:12.179960 22038 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 01:14:12.180725 22053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:14:12.180945 22038 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:14:12.181038 22038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790384993574762-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790384993574762-392-0/minicluster-data/master-0/wal
uuid: "635a2b70804f4c29bb3f0903e27dd56a"
format_stamp: "Formatted at 2026-09-26 01:14:12 on dist-test-slave-7b22"
server_key: "0e2f00f7cbe8391bc2adbced553be94b"
server_key_iv: "4610c461dfede924a0c5da44880cc5d2"
server_key_version: "encryptionkey@0"
I20260926 01:14:12.181102 22038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790384993574762-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790384993574762-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790384993574762-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:14:12.193306 22038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:14:12.195518 22038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:14:12.195663 22038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:14:12.199398 22038 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40939
I20260926 01:14:12.199478 22105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40939 every 8 connection(s)
I20260926 01:14:12.199769 22038 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
I20260926 01:14:12.200574 22106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:14:12.202890 22106 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 635a2b70804f4c29bb3f0903e27dd56a: Bootstrap starting.
I20260926 01:14:12.203425 22106 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 635a2b70804f4c29bb3f0903e27dd56a: Neither blocks nor log segments found. Creating new log.
I20260926 01:14:12.203645 22106 log.cc:824] T 00000000000000000000000000000000 P 635a2b70804f4c29bb3f0903e27dd56a: Log is configured to *not* fsync() on all Append() calls
I20260926 01:14:12.204289 22106 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 635a2b70804f4c29bb3f0903e27dd56a: No bootstrap required, opened a new log
I20260926 01:14:12.205858 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 22038
I20260926 01:14:12.205967 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790384993574762-392-0/minicluster-data/master-0/wal/instance
I20260926 01:14:12.206121 392 external_mini_cluster.cc:1428] Setting key 24052adde1c21331e88796c77f11c361
I20260926 01:14:12.206025 22106 raft_consensus.cc:348] T 00000000000000000000000000000000 P 635a2b70804f4c29bb3f0903e27dd56a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "635a2b70804f4c29bb3f0903e27dd56a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40939 } }
I20260926 01:14:12.206172 22106 raft_consensus.cc:374] T 00000000000000000000000000000000 P 635a2b70804f4c29bb3f0903e27dd56a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:14:12.206214 22106 raft_consensus.cc:729] T 00000000000000000000000000000000 P 635a2b70804f4c29bb3f0903e27dd56a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 635a2b70804f4c29bb3f0903e27dd56a, State: Initialized, Role: FOLLOWER
I20260926 01:14:12.206367 22106 consensus_queue.cc:260] T 00000000000000000000000000000000 P 635a2b70804f4c29bb3f0903e27dd56a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "635a2b70804f4c29bb3f0903e27dd56a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40939 } }
I20260926 01:14:12.206451 22106 raft_consensus.cc:388] T 00000000000000000000000000000000 P 635a2b70804f4c29bb3f0903e27dd56a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:14:12.206697 22106 raft_consensus.cc:482] T 00000000000000000000000000000000 P 635a2b70804f4c29bb3f0903e27dd56a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:14:12.206796 22106 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 635a2b70804f4c29bb3f0903e27dd56a [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:14:12.207831 22106 raft_consensus.cc:504] T 00000000000000000000000000000000 P 635a2b70804f4c29bb3f0903e27dd56a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "635a2b70804f4c29bb3f0903e27dd56a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40939 } }
I20260926 01:14:12.207926 22106 leader_election.cc:302] T 00000000000000000000000000000000 P 635a2b70804f4c29bb3f0903e27dd56a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 635a2b70804f4c29bb3f0903e27dd56a; no voters:
I20260926 01:14:12.208086 22106 leader_election.cc:288] T 00000000000000000000000000000000 P 635a2b70804f4c29bb3f0903e27dd56a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:14:12.208112 22111 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 635a2b70804f4c29bb3f0903e27dd56a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:14:12.208256 22111 raft_consensus.cc:686] T 00000000000000000000000000000000 P 635a2b70804f4c29bb3f0903e27dd56a [term 1 LEADER]: Becoming Leader. State: Replica: 635a2b70804f4c29bb3f0903e27dd56a, State: Running, Role: LEADER
I20260926 01:14:12.208352 22106 sys_catalog.cc:564] T 00000000000000000000000000000000 P 635a2b70804f4c29bb3f0903e27dd56a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:14:12.208386 22111 consensus_queue.cc:237] T 00000000000000000000000000000000 P 635a2b70804f4c29bb3f0903e27dd56a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "635a2b70804f4c29bb3f0903e27dd56a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40939 } }
I20260926 01:14:12.208768 22111 sys_catalog.cc:455] T 00000000000000000000000000000000 P 635a2b70804f4c29bb3f0903e27dd56a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 635a2b70804f4c29bb3f0903e27dd56a. Latest consensus state: current_term: 1 leader_uuid: "635a2b70804f4c29bb3f0903e27dd56a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "635a2b70804f4c29bb3f0903e27dd56a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40939 } } }
I20260926 01:14:12.208911 22111 sys_catalog.cc:458] T 00000000000000000000000000000000 P 635a2b70804f4c29bb3f0903e27dd56a [sys.catalog]: This master's current role is: LEADER
I20260926 01:14:12.209149 22120 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:14:12.209669 22120 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:14:12.209651 22113 sys_catalog.cc:455] T 00000000000000000000000000000000 P 635a2b70804f4c29bb3f0903e27dd56a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "635a2b70804f4c29bb3f0903e27dd56a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "635a2b70804f4c29bb3f0903e27dd56a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40939 } } }
I20260926 01:14:12.209956 22113 sys_catalog.cc:458] T 00000000000000000000000000000000 P 635a2b70804f4c29bb3f0903e27dd56a [sys.catalog]: This master's current role is: LEADER
I20260926 01:14:12.211319 22120 catalog_manager.cc:1347] Generated new cluster ID: 42ba2b3676a441bdac234a6829007957
I20260926 01:14:12.211380 22120 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 01:14:12.214603 22110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:14:12.206541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52439 (local address 127.0.98.62:40939)
0926 01:14:12.206735 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 01:14:12.206746 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 01:14:12.206789 (+ 43us) server_negotiation.cc:407] Connection header received
0926 01:14:12.206952 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:14:12.206958 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:14:12.207005 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:14:12.207114 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:14:12.207354 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:14:12.208433 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:14:12.209452 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:14:12.210088 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:14:12.210189 (+ 101us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:14:12.210933 (+ 744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:14:12.210937 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:14:12.210938 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:14:12.211120 (+ 182us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:14:12.211151 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:14:12.211162 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:14:12.211235 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 01:14:12.214336 (+ 3101us) server_negotiation.cc:299] Negotiation successful
0926 01:14:12.214369 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":87,"threads_started":1}
I20260926 01:14:12.215641 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 01:14:12.220670 22110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:14:12.217165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36671 (local address 127.0.98.62:40939)
0926 01:14:12.217203 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 01:14:12.217205 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:14:12.217354 (+ 149us) server_negotiation.cc:407] Connection header received
0926 01:14:12.217411 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:14:12.217412 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:14:12.217424 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:14:12.217474 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:14:12.217605 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:14:12.218401 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:14:12.219371 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:14:12.219693 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:14:12.219768 (+ 75us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:14:12.220424 (+ 656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:14:12.220426 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:14:12.220426 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:14:12.220431 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:14:12.220446 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:14:12.220450 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:14:12.220508 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 01:14:12.220574 (+ 66us) server_negotiation.cc:299] Negotiation successful
0926 01:14:12.220586 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 01:14:12.222532 22120 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:14:12.223026 22120 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:14:12.231810 22120 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 635a2b70804f4c29bb3f0903e27dd56a: Generated new TSK 0
I20260926 01:14:12.232025 22120 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 01:14:12.236809 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 22038
W20260926 01:14:12.240767 22023 connection.cc:535] client connection to 127.0.98.62:40939 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:40939): Connection reset by peer (error 104)
2026-09-26T01:14:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1234 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790384993574762-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 01:14:12 dist-test-slave-7b22 krb5kdc[22137](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 01:14:12 dist-test-slave-7b22 krb5kdc[22137](info): set up 2 sockets
Sep 26 01:14:12 dist-test-slave-7b22 krb5kdc[22137](info): commencing operation
krb5kdc: starting...
W20260926 01:14:13.213793 392 mini_kdc.cc:121] Time spent starting KDC: real 0.959s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790384993574762-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790384993574762-392-0/krb5kdc/test-user.keytab.
Sep 26 01:14:13 dist-test-slave-7b22 krb5kdc[22137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385253, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T01:14:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:14:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790384993574762-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790384993574762-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 01:14:13.344964 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790384993574762-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790384993574762-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790384993574762-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43377
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790384993574762-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43377
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:14:13.432927 22150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:14:13.433164 22150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:14:13.433198 22150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:14:13.434509 22150 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:14:13.434557 22150 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:14:13.434577 22150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:14:13.434594 22150 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:14:13.434610 22150 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:14:13.436185 22150 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790384993574762-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790384993574762-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43377
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790384993574762-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43377
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22150
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790384993574762-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:14:13.436439 22150 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:14:13.436648 22150 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:14:13.441629 22156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:14:13.441838 22150 server_base.cc:1034] running on GCE node
W20260926 01:14:13.441633 22158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:14:13.441633 22155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:14:13.442274 22150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:14:13.442569 22150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:14:13.443729 22150 hybrid_clock.cc:648] HybridClock initialized: now 1790385253443711 us; error 36 us; skew 500 ppm
Sep 26 01:14:13 dist-test-slave-7b22 krb5kdc[22137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385253, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:14:13.446404 22150 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 01:14:13.446837 22150 webserver.cc:460] Webserver started at http://127.0.98.62:35485/ using document root <none> and password file <none>
I20260926 01:14:13.447038 22150 fs_manager.cc:359] Metadata directory not provided
I20260926 01:14:13.447080 22150 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:14:13.447190 22150 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:14:13.448305 22150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790384993574762-392-0/minicluster-data/master-0/data/instance:
uuid: "fa4774ce187e412e9a973c636b391feb"
format_stamp: "Formatted at 2026-09-26 01:14:13 on dist-test-slave-7b22"
server_key: "aa25575e1cbf39f2e2ba86f9606d9f50"
server_key_iv: "b049cf97ea2cc58c22b14cb2fd0002d5"
server_key_version: "encryptionkey@0"
I20260926 01:14:13.448644 22150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790384993574762-392-0/minicluster-data/master-0/wal/instance:
uuid: "fa4774ce187e412e9a973c636b391feb"
format_stamp: "Formatted at 2026-09-26 01:14:13 on dist-test-slave-7b22"
server_key: "aa25575e1cbf39f2e2ba86f9606d9f50"
server_key_iv: "b049cf97ea2cc58c22b14cb2fd0002d5"
server_key_version: "encryptionkey@0"
I20260926 01:14:13.449977 22150 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 01:14:13.451068 22165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:14:13.451292 22150 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:14:13.451385 22150 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790384993574762-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790384993574762-392-0/minicluster-data/master-0/wal
uuid: "fa4774ce187e412e9a973c636b391feb"
format_stamp: "Formatted at 2026-09-26 01:14:13 on dist-test-slave-7b22"
server_key: "aa25575e1cbf39f2e2ba86f9606d9f50"
server_key_iv: "b049cf97ea2cc58c22b14cb2fd0002d5"
server_key_version: "encryptionkey@0"
I20260926 01:14:13.451493 22150 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790384993574762-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790384993574762-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790384993574762-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:14:13.455027 22150 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:14:13.457168 22150 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:14:13.457283 22150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:14:13.461158 22150 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43377
I20260926 01:14:13.461225 22217 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43377 every 8 connection(s)
I20260926 01:14:13.461541 22150 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
I20260926 01:14:13.462988 22218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:14:13.466310 22218 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa4774ce187e412e9a973c636b391feb: Bootstrap starting.
I20260926 01:14:13.466920 22218 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fa4774ce187e412e9a973c636b391feb: Neither blocks nor log segments found. Creating new log.
I20260926 01:14:13.467175 22218 log.cc:824] T 00000000000000000000000000000000 P fa4774ce187e412e9a973c636b391feb: Log is configured to *not* fsync() on all Append() calls
I20260926 01:14:13.468019 22218 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa4774ce187e412e9a973c636b391feb: No bootstrap required, opened a new log
I20260926 01:14:13.469696 22218 raft_consensus.cc:348] T 00000000000000000000000000000000 P fa4774ce187e412e9a973c636b391feb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa4774ce187e412e9a973c636b391feb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43377 } }
I20260926 01:14:13.469915 22218 raft_consensus.cc:374] T 00000000000000000000000000000000 P fa4774ce187e412e9a973c636b391feb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:14:13.469962 22218 raft_consensus.cc:729] T 00000000000000000000000000000000 P fa4774ce187e412e9a973c636b391feb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa4774ce187e412e9a973c636b391feb, State: Initialized, Role: FOLLOWER
I20260926 01:14:13.470105 22218 consensus_queue.cc:260] T 00000000000000000000000000000000 P fa4774ce187e412e9a973c636b391feb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa4774ce187e412e9a973c636b391feb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43377 } }
I20260926 01:14:13.470173 22218 raft_consensus.cc:388] T 00000000000000000000000000000000 P fa4774ce187e412e9a973c636b391feb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:14:13.470211 22218 raft_consensus.cc:482] T 00000000000000000000000000000000 P fa4774ce187e412e9a973c636b391feb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:14:13.470258 22218 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fa4774ce187e412e9a973c636b391feb [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:14:13.470407 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 22150
I20260926 01:14:13.470515 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790384993574762-392-0/minicluster-data/master-0/wal/instance
I20260926 01:14:13.470681 392 external_mini_cluster.cc:1428] Setting key 800f7d74369513d8c890acd34a47b57a
I20260926 01:14:13.470942 22218 raft_consensus.cc:504] T 00000000000000000000000000000000 P fa4774ce187e412e9a973c636b391feb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa4774ce187e412e9a973c636b391feb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43377 } }
I20260926 01:14:13.471041 22218 leader_election.cc:302] T 00000000000000000000000000000000 P fa4774ce187e412e9a973c636b391feb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fa4774ce187e412e9a973c636b391feb; no voters:
I20260926 01:14:13.471428 22218 leader_election.cc:288] T 00000000000000000000000000000000 P fa4774ce187e412e9a973c636b391feb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:14:13.471481 22223 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fa4774ce187e412e9a973c636b391feb [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:14:13.471668 22223 raft_consensus.cc:686] T 00000000000000000000000000000000 P fa4774ce187e412e9a973c636b391feb [term 1 LEADER]: Becoming Leader. State: Replica: fa4774ce187e412e9a973c636b391feb, State: Running, Role: LEADER
I20260926 01:14:13.471697 22218 sys_catalog.cc:564] T 00000000000000000000000000000000 P fa4774ce187e412e9a973c636b391feb [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:14:13.472569 22223 consensus_queue.cc:237] T 00000000000000000000000000000000 P fa4774ce187e412e9a973c636b391feb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa4774ce187e412e9a973c636b391feb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43377 } }
I20260926 01:14:13.472843 22222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:14:13.471114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48817 (local address 127.0.98.62:43377)
0926 01:14:13.471292 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 01:14:13.471300 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 01:14:13.471339 (+ 39us) server_negotiation.cc:407] Connection header received
0926 01:14:13.471476 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:14:13.471484 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:14:13.471519 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:14:13.471673 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:14:13.472346 (+ 673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:14:13.472350 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:14:13.472350 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:14:13.472354 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:14:13.472404 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:14:13.472421 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:14:13.472494 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 01:14:13.472611 (+ 117us) server_negotiation.cc:299] Negotiation successful
0926 01:14:13.472657 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":93,"threads_started":1}
I20260926 01:14:13.473276 22224 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa4774ce187e412e9a973c636b391feb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fa4774ce187e412e9a973c636b391feb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa4774ce187e412e9a973c636b391feb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43377 } } }
I20260926 01:14:13.473362 22224 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa4774ce187e412e9a973c636b391feb [sys.catalog]: This master's current role is: LEADER
I20260926 01:14:13.473686 22225 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa4774ce187e412e9a973c636b391feb [sys.catalog]: SysCatalogTable state changed. Reason: New leader fa4774ce187e412e9a973c636b391feb. Latest consensus state: current_term: 1 leader_uuid: "fa4774ce187e412e9a973c636b391feb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa4774ce187e412e9a973c636b391feb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43377 } } }
I20260926 01:14:13.473784 22225 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa4774ce187e412e9a973c636b391feb [sys.catalog]: This master's current role is: LEADER
I20260926 01:14:13.473946 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260926 01:14:13.474074 22237 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fa4774ce187e412e9a973c636b391feb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 01:14:13.474118 22237 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 01:14:13.474175 22238 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:14:13.474354 22238 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:14:13.476214 22238 catalog_manager.cc:1347] Generated new cluster ID: 2f0a891a61fa4f018187258def946d3a
I20260926 01:14:13.476265 22238 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 01:14:13.476593 22222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:14:13.475581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43281 (local address 127.0.98.62:43377)
0926 01:14:13.475612 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0926 01:14:13.475614 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:14:13.475637 (+ 23us) server_negotiation.cc:407] Connection header received
0926 01:14:13.475657 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:14:13.475658 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:14:13.475662 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:14:13.475704 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:14:13.476320 (+ 616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:14:13.476321 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:14:13.476322 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:14:13.476326 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:14:13.476344 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:14:13.476350 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:14:13.476415 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 01:14:13.476517 (+ 102us) server_negotiation.cc:299] Negotiation successful
0926 01:14:13.476532 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260926 01:14:13.483356 22238 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:14:13.483659 22238 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:14:13.487421 22238 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fa4774ce187e412e9a973c636b391feb: Generated new TSK 0
I20260926 01:14:13.487629 22238 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 01:14:13.493706 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 22150
2026-09-26T01:14:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1257 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790384993574762-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 01:14:13 dist-test-slave-7b22 krb5kdc[22249](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 01:14:13 dist-test-slave-7b22 krb5kdc[22249](info): set up 2 sockets
Sep 26 01:14:13 dist-test-slave-7b22 krb5kdc[22249](info): commencing operation
krb5kdc: starting...
W20260926 01:14:14.470341 392 mini_kdc.cc:121] Time spent starting KDC: real 0.959s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790384993574762-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790384993574762-392-0/krb5kdc/test-user.keytab.
Sep 26 01:14:14 dist-test-slave-7b22 krb5kdc[22249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385254, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T01:14:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:14:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790384993574762-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790384993574762-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 01:14:14.597388 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790384993574762-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790384993574762-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790384993574762-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41381
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790384993574762-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41381
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:14:14.682111 22262 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:14:14.682361 22262 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:14:14.682401 22262 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:14:14.684027 22262 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:14:14.684092 22262 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:14:14.684114 22262 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:14:14.684145 22262 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:14:14.684170 22262 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:14:14.685935 22262 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790384993574762-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790384993574762-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41381
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790384993574762-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41381
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22262
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790384993574762-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:14:14.686164 22262 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:14:14.686371 22262 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:14:14.691418 22270 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:14:14.691411 22267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:14:14.691461 22262 server_base.cc:1034] running on GCE node
W20260926 01:14:14.691461 22268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:14:14.691920 22262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:14:14.692286 22262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:14:14.693454 22262 hybrid_clock.cc:648] HybridClock initialized: now 1790385254693431 us; error 33 us; skew 500 ppm
Sep 26 01:14:14 dist-test-slave-7b22 krb5kdc[22249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385254, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:14:14.696144 22262 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 01:14:14.696615 22262 webserver.cc:460] Webserver started at http://127.0.98.62:46339/ using document root <none> and password file <none>
I20260926 01:14:14.696836 22262 fs_manager.cc:359] Metadata directory not provided
I20260926 01:14:14.696879 22262 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:14:14.697013 22262 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:14:14.698151 22262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790384993574762-392-0/minicluster-data/master-0/data/instance:
uuid: "dcb1a18b939b45c7884cbd672dd108b8"
format_stamp: "Formatted at 2026-09-26 01:14:14 on dist-test-slave-7b22"
server_key: "5cb4cf7e230517424f306c726db66896"
server_key_iv: "cb611ee29ebec74b621265d332b2bdd4"
server_key_version: "encryptionkey@0"
I20260926 01:14:14.698482 22262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790384993574762-392-0/minicluster-data/master-0/wal/instance:
uuid: "dcb1a18b939b45c7884cbd672dd108b8"
format_stamp: "Formatted at 2026-09-26 01:14:14 on dist-test-slave-7b22"
server_key: "5cb4cf7e230517424f306c726db66896"
server_key_iv: "cb611ee29ebec74b621265d332b2bdd4"
server_key_version: "encryptionkey@0"
I20260926 01:14:14.699754 22262 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 01:14:14.700667 22277 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:14:14.701013 22262 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:14:14.701118 22262 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790384993574762-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790384993574762-392-0/minicluster-data/master-0/wal
uuid: "dcb1a18b939b45c7884cbd672dd108b8"
format_stamp: "Formatted at 2026-09-26 01:14:14 on dist-test-slave-7b22"
server_key: "5cb4cf7e230517424f306c726db66896"
server_key_iv: "cb611ee29ebec74b621265d332b2bdd4"
server_key_version: "encryptionkey@0"
I20260926 01:14:14.701212 22262 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790384993574762-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790384993574762-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790384993574762-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:14:14.716490 22262 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:14:14.718855 22262 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:14:14.718997 22262 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:14:14.723718 22262 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41381
I20260926 01:14:14.723798 22329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41381 every 8 connection(s)
I20260926 01:14:14.724179 22262 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
I20260926 01:14:14.725471 22330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:14:14.728268 22330 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dcb1a18b939b45c7884cbd672dd108b8: Bootstrap starting.
I20260926 01:14:14.728909 22330 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dcb1a18b939b45c7884cbd672dd108b8: Neither blocks nor log segments found. Creating new log.
I20260926 01:14:14.729168 22330 log.cc:824] T 00000000000000000000000000000000 P dcb1a18b939b45c7884cbd672dd108b8: Log is configured to *not* fsync() on all Append() calls
I20260926 01:14:14.730362 22330 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dcb1a18b939b45c7884cbd672dd108b8: No bootstrap required, opened a new log
I20260926 01:14:14.732402 22330 raft_consensus.cc:348] T 00000000000000000000000000000000 P dcb1a18b939b45c7884cbd672dd108b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcb1a18b939b45c7884cbd672dd108b8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41381 } }
I20260926 01:14:14.732592 22330 raft_consensus.cc:374] T 00000000000000000000000000000000 P dcb1a18b939b45c7884cbd672dd108b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:14:14.732625 22330 raft_consensus.cc:729] T 00000000000000000000000000000000 P dcb1a18b939b45c7884cbd672dd108b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcb1a18b939b45c7884cbd672dd108b8, State: Initialized, Role: FOLLOWER
I20260926 01:14:14.732761 22330 consensus_queue.cc:260] T 00000000000000000000000000000000 P dcb1a18b939b45c7884cbd672dd108b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcb1a18b939b45c7884cbd672dd108b8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41381 } }
I20260926 01:14:14.732847 22330 raft_consensus.cc:388] T 00000000000000000000000000000000 P dcb1a18b939b45c7884cbd672dd108b8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:14:14.732873 22330 raft_consensus.cc:482] T 00000000000000000000000000000000 P dcb1a18b939b45c7884cbd672dd108b8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:14:14.732918 22330 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dcb1a18b939b45c7884cbd672dd108b8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:14:14.733071 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 22262
I20260926 01:14:14.733180 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790384993574762-392-0/minicluster-data/master-0/wal/instance
I20260926 01:14:14.733348 392 external_mini_cluster.cc:1428] Setting key 769ee554092f3d68651a4658479c42bc
I20260926 01:14:14.733716 22330 raft_consensus.cc:504] T 00000000000000000000000000000000 P dcb1a18b939b45c7884cbd672dd108b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcb1a18b939b45c7884cbd672dd108b8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41381 } }
I20260926 01:14:14.733866 22330 leader_election.cc:302] T 00000000000000000000000000000000 P dcb1a18b939b45c7884cbd672dd108b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dcb1a18b939b45c7884cbd672dd108b8; no voters:
I20260926 01:14:14.734043 22330 leader_election.cc:288] T 00000000000000000000000000000000 P dcb1a18b939b45c7884cbd672dd108b8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:14:14.734113 22335 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dcb1a18b939b45c7884cbd672dd108b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:14:14.734257 22335 raft_consensus.cc:686] T 00000000000000000000000000000000 P dcb1a18b939b45c7884cbd672dd108b8 [term 1 LEADER]: Becoming Leader. State: Replica: dcb1a18b939b45c7884cbd672dd108b8, State: Running, Role: LEADER
I20260926 01:14:14.734385 22330 sys_catalog.cc:564] T 00000000000000000000000000000000 P dcb1a18b939b45c7884cbd672dd108b8 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:14:14.734355 22335 consensus_queue.cc:237] T 00000000000000000000000000000000 P dcb1a18b939b45c7884cbd672dd108b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcb1a18b939b45c7884cbd672dd108b8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41381 } }
I20260926 01:14:14.734851 22337 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcb1a18b939b45c7884cbd672dd108b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dcb1a18b939b45c7884cbd672dd108b8. Latest consensus state: current_term: 1 leader_uuid: "dcb1a18b939b45c7884cbd672dd108b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcb1a18b939b45c7884cbd672dd108b8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41381 } } }
I20260926 01:14:14.734985 22337 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcb1a18b939b45c7884cbd672dd108b8 [sys.catalog]: This master's current role is: LEADER
I20260926 01:14:14.736227 22336 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcb1a18b939b45c7884cbd672dd108b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dcb1a18b939b45c7884cbd672dd108b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcb1a18b939b45c7884cbd672dd108b8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41381 } } }
I20260926 01:14:14.736368 22336 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcb1a18b939b45c7884cbd672dd108b8 [sys.catalog]: This master's current role is: LEADER
W20260926 01:14:14.736459 22350 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dcb1a18b939b45c7884cbd672dd108b8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 01:14:14.736526 22350 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 01:14:14.736603 22344 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:14:14.736776 22344 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:14:14.738433 22344 catalog_manager.cc:1347] Generated new cluster ID: 2357598c05804a578aa89ea5d620bb9e
I20260926 01:14:14.738492 22344 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 01:14:14 dist-test-slave-7b22 krb5kdc[22249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385254, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:14:14.746711 22334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:14:14.733883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38633 (local address 127.0.98.62:41381)
0926 01:14:14.734096 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0926 01:14:14.734107 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 01:14:14.734153 (+ 46us) server_negotiation.cc:407] Connection header received
0926 01:14:14.734302 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:14:14.734310 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:14:14.734378 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:14:14.734521 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:14:14.735866 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:14:14.737209 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:14:14.738212 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:14:14.738789 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:14:14.738894 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:14:14.742355 (+ 3461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:14:14.742361 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:14:14.742362 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:14:14.742378 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:14:14.744714 (+ 2336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:14:14.745238 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:14:14.745242 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:14:14.745242 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:14:14.745396 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:14:14.745632 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:14:14.745633 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:14:14.745633 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:14:14.746049 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:14:14.746227 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0926 01:14:14.746314 (+ 87us) server_negotiation.cc:299] Negotiation successful
0926 01:14:14.746470 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":101,"threads_started":1}
I20260926 01:14:14.747884 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 01:14:14.752858 22344 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:14:14.753665 22344 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:14:14.757292 22334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:14:14.749454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36769 (local address 127.0.98.62:41381)
0926 01:14:14.749498 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 01:14:14.749499 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 01:14:14.749556 (+ 57us) server_negotiation.cc:407] Connection header received
0926 01:14:14.749619 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:14:14.749620 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:14:14.749630 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:14:14.749703 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:14:14.750426 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:14:14.751198 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:14:14.752280 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:14:14.752529 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:14:14.752602 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:14:14.754079 (+ 1477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:14:14.754081 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:14:14.754081 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:14:14.754090 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:14:14.756105 (+ 2015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:14:14.756510 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:14:14.756511 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:14:14.756512 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:14:14.756571 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:14:14.756747 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:14:14.756748 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:14:14.756748 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:14:14.756869 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:14:14.756976 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 01:14:14.757093 (+ 117us) server_negotiation.cc:299] Negotiation successful
0926 01:14:14.757205 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 01:14:14.762151 22344 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dcb1a18b939b45c7884cbd672dd108b8: Generated new TSK 0
I20260926 01:14:14.762352 22344 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 01:14:14.770622 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 22262
2026-09-26T01:14:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1277 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790384993574762-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 01:14:14 dist-test-slave-7b22 krb5kdc[22361](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 01:14:14 dist-test-slave-7b22 krb5kdc[22361](info): set up 2 sockets
Sep 26 01:14:14 dist-test-slave-7b22 krb5kdc[22361](info): commencing operation
krb5kdc: starting...
W20260926 01:14:15.734622 392 mini_kdc.cc:121] Time spent starting KDC: real 0.945s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790384993574762-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790384993574762-392-0/krb5kdc/test-user.keytab.
Sep 26 01:14:15 dist-test-slave-7b22 krb5kdc[22361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385255, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T01:14:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:14:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790384993574762-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790384993574762-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 01:14:15.857331 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790384993574762-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790384993574762-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790384993574762-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35815
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790384993574762-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35815
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:14:15.940271 22374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:14:15.940483 22374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:14:15.940507 22374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:14:15.941639 22374 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:14:15.941679 22374 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:14:15.941691 22374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:14:15.941702 22374 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:14:15.941712 22374 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:14:15.943239 22374 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790384993574762-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790384993574762-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35815
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790384993574762-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35815
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22374
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790384993574762-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:14:15.943437 22374 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:14:15.943617 22374 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:14:15.948258 22382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:14:15.948238 22379 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:14:15.948390 22374 server_base.cc:1034] running on GCE node
W20260926 01:14:15.948241 22380 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:14:15.948769 22374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:14:15.949047 22374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:14:15.950178 22374 hybrid_clock.cc:648] HybridClock initialized: now 1790385255950157 us; error 41 us; skew 500 ppm
Sep 26 01:14:15 dist-test-slave-7b22 krb5kdc[22361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385255, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:14:15.952497 22374 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 01:14:15.952979 22374 webserver.cc:460] Webserver started at http://127.0.98.62:46019/ using document root <none> and password file <none>
I20260926 01:14:15.953173 22374 fs_manager.cc:359] Metadata directory not provided
I20260926 01:14:15.953219 22374 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:14:15.953342 22374 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:14:15.954429 22374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790384993574762-392-0/minicluster-data/master-0/data/instance:
uuid: "85acd373119942d79a45125f271873ee"
format_stamp: "Formatted at 2026-09-26 01:14:15 on dist-test-slave-7b22"
server_key: "876442422f33f6f9d15d0550f142ece7"
server_key_iv: "9f38549e1f9f78c58dc08c059ee6849e"
server_key_version: "encryptionkey@0"
I20260926 01:14:15.954737 22374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790384993574762-392-0/minicluster-data/master-0/wal/instance:
uuid: "85acd373119942d79a45125f271873ee"
format_stamp: "Formatted at 2026-09-26 01:14:15 on dist-test-slave-7b22"
server_key: "876442422f33f6f9d15d0550f142ece7"
server_key_iv: "9f38549e1f9f78c58dc08c059ee6849e"
server_key_version: "encryptionkey@0"
I20260926 01:14:15.955973 22374 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 01:14:15.956981 22389 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:14:15.957293 22374 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:14:15.957386 22374 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790384993574762-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790384993574762-392-0/minicluster-data/master-0/wal
uuid: "85acd373119942d79a45125f271873ee"
format_stamp: "Formatted at 2026-09-26 01:14:15 on dist-test-slave-7b22"
server_key: "876442422f33f6f9d15d0550f142ece7"
server_key_iv: "9f38549e1f9f78c58dc08c059ee6849e"
server_key_version: "encryptionkey@0"
I20260926 01:14:15.957460 22374 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790384993574762-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790384993574762-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790384993574762-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:14:15.965991 22374 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:14:15.967700 22374 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:14:15.967832 22374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:14:15.971588 22374 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35815
I20260926 01:14:15.971622 22441 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35815 every 8 connection(s)
I20260926 01:14:15.971961 22374 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
I20260926 01:14:15.972366 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 22374
I20260926 01:14:15.972466 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790384993574762-392-0/minicluster-data/master-0/wal/instance
I20260926 01:14:15.972638 392 external_mini_cluster.cc:1428] Setting key ad4e68680519dcd3fb772f7adb68c6cd
I20260926 01:14:15.972937 22442 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:14:15.976310 22442 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85acd373119942d79a45125f271873ee: Bootstrap starting.
I20260926 01:14:15.976958 22442 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 85acd373119942d79a45125f271873ee: Neither blocks nor log segments found. Creating new log.
I20260926 01:14:15.977259 22442 log.cc:824] T 00000000000000000000000000000000 P 85acd373119942d79a45125f271873ee: Log is configured to *not* fsync() on all Append() calls
I20260926 01:14:15.978073 22442 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85acd373119942d79a45125f271873ee: No bootstrap required, opened a new log
I20260926 01:14:15.978806 22445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:14:15.973178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59657 (local address 127.0.98.62:35815)
0926 01:14:15.973385 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 01:14:15.973394 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 01:14:15.973426 (+ 32us) server_negotiation.cc:407] Connection header received
0926 01:14:15.974095 (+ 669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:14:15.974104 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:14:15.974169 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:14:15.974321 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:14:15.974542 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:14:15.975858 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:14:15.976677 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:14:15.977284 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:14:15.977393 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:14:15.978092 (+ 699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:14:15.978096 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:14:15.978097 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:14:15.978305 (+ 208us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:14:15.978343 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:14:15.978358 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:14:15.978423 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 01:14:15.978537 (+ 114us) server_negotiation.cc:299] Negotiation successful
0926 01:14:15.978604 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":93,"threads_started":1}
I20260926 01:14:15.979939 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 01:14:15.980166 22442 raft_consensus.cc:348] T 00000000000000000000000000000000 P 85acd373119942d79a45125f271873ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85acd373119942d79a45125f271873ee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35815 } }
I20260926 01:14:15.980326 22442 raft_consensus.cc:374] T 00000000000000000000000000000000 P 85acd373119942d79a45125f271873ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:14:15.980363 22442 raft_consensus.cc:729] T 00000000000000000000000000000000 P 85acd373119942d79a45125f271873ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85acd373119942d79a45125f271873ee, State: Initialized, Role: FOLLOWER
I20260926 01:14:15.980482 22442 consensus_queue.cc:260] T 00000000000000000000000000000000 P 85acd373119942d79a45125f271873ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85acd373119942d79a45125f271873ee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35815 } }
I20260926 01:14:15.980556 22442 raft_consensus.cc:388] T 00000000000000000000000000000000 P 85acd373119942d79a45125f271873ee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:14:15.980583 22442 raft_consensus.cc:482] T 00000000000000000000000000000000 P 85acd373119942d79a45125f271873ee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:14:15.980619 22442 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 85acd373119942d79a45125f271873ee [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:14:15.981333 22442 raft_consensus.cc:504] T 00000000000000000000000000000000 P 85acd373119942d79a45125f271873ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85acd373119942d79a45125f271873ee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35815 } }
I20260926 01:14:15.981424 22442 leader_election.cc:302] T 00000000000000000000000000000000 P 85acd373119942d79a45125f271873ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 85acd373119942d79a45125f271873ee; no voters:
I20260926 01:14:15.981561 22442 leader_election.cc:288] T 00000000000000000000000000000000 P 85acd373119942d79a45125f271873ee [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:14:15.981613 22451 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 85acd373119942d79a45125f271873ee [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:14:15.981806 22451 raft_consensus.cc:686] T 00000000000000000000000000000000 P 85acd373119942d79a45125f271873ee [term 1 LEADER]: Becoming Leader. State: Replica: 85acd373119942d79a45125f271873ee, State: Running, Role: LEADER
I20260926 01:14:15.981892 22442 sys_catalog.cc:564] T 00000000000000000000000000000000 P 85acd373119942d79a45125f271873ee [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:14:15.981974 22451 consensus_queue.cc:237] T 00000000000000000000000000000000 P 85acd373119942d79a45125f271873ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85acd373119942d79a45125f271873ee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35815 } }
I20260926 01:14:15.982364 22454 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85acd373119942d79a45125f271873ee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "85acd373119942d79a45125f271873ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85acd373119942d79a45125f271873ee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35815 } } }
I20260926 01:14:15.982470 22454 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85acd373119942d79a45125f271873ee [sys.catalog]: This master's current role is: LEADER
I20260926 01:14:15.982353 22451 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85acd373119942d79a45125f271873ee [sys.catalog]: SysCatalogTable state changed. Reason: New leader 85acd373119942d79a45125f271873ee. Latest consensus state: current_term: 1 leader_uuid: "85acd373119942d79a45125f271873ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85acd373119942d79a45125f271873ee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35815 } } }
I20260926 01:14:15.982717 22451 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85acd373119942d79a45125f271873ee [sys.catalog]: This master's current role is: LEADER
I20260926 01:14:15.983008 22461 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:14:15.983609 22461 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:14:15.984933 22445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:14:15.981662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59691 (local address 127.0.98.62:35815)
0926 01:14:15.981695 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0926 01:14:15.981697 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:14:15.981706 (+ 9us) server_negotiation.cc:407] Connection header received
0926 01:14:15.981747 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:14:15.981747 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:14:15.981758 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:14:15.981876 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:14:15.982025 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:14:15.982710 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:14:15.983716 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:14:15.983996 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:14:15.984060 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:14:15.984693 (+ 633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:14:15.984694 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:14:15.984694 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:14:15.984699 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:14:15.984713 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:14:15.984717 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:14:15.984771 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 01:14:15.984834 (+ 63us) server_negotiation.cc:299] Negotiation successful
0926 01:14:15.984857 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 01:14:15.984936 22461 catalog_manager.cc:1347] Generated new cluster ID: 719807c04bd8478a9776cc90dea4cbcd
I20260926 01:14:15.985010 22461 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 01:14:16.005452 22461 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:14:16.006410 22461 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:14:16.012560 22461 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 85acd373119942d79a45125f271873ee: Generated new TSK 0
I20260926 01:14:16.012802 22461 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 01:14:16.021919 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 22374
2026-09-26T01:14:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1250 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790384993574762-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 01:14:16 dist-test-slave-7b22 krb5kdc[22473](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 01:14:16 dist-test-slave-7b22 krb5kdc[22473](info): set up 2 sockets
Sep 26 01:14:16 dist-test-slave-7b22 krb5kdc[22473](info): commencing operation
krb5kdc: starting...
W20260926 01:14:16.986063 392 mini_kdc.cc:121] Time spent starting KDC: real 0.946s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790384993574762-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790384993574762-392-0/krb5kdc/test-user.keytab.
Sep 26 01:14:17 dist-test-slave-7b22 krb5kdc[22473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385257, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T01:14:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:14:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790384993574762-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790384993574762-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 01:14:17.107257 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790384993574762-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790384993574762-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790384993574762-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42999
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790384993574762-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42999
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:14:17.189075 22486 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:14:17.189317 22486 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:14:17.189352 22486 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:14:17.190698 22486 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:14:17.190757 22486 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:14:17.190775 22486 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:14:17.190793 22486 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:14:17.190806 22486 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:14:17.192446 22486 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790384993574762-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790384993574762-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42999
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790384993574762-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42999
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22486
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790384993574762-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:14:17.192713 22486 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:14:17.192945 22486 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:14:17.198068 22494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:14:17.198197 22492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:14:17.198083 22486 server_base.cc:1034] running on GCE node
W20260926 01:14:17.198077 22491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:14:17.198598 22486 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:14:17.198943 22486 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:14:17.200096 22486 hybrid_clock.cc:648] HybridClock initialized: now 1790385257200082 us; error 34 us; skew 500 ppm
Sep 26 01:14:17 dist-test-slave-7b22 krb5kdc[22473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385257, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:14:17.202646 22486 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 01:14:17.203118 22486 webserver.cc:460] Webserver started at http://127.0.98.62:42073/ using document root <none> and password file <none>
I20260926 01:14:17.203318 22486 fs_manager.cc:359] Metadata directory not provided
I20260926 01:14:17.203363 22486 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:14:17.203455 22486 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:14:17.204463 22486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790384993574762-392-0/minicluster-data/master-0/data/instance:
uuid: "1ff16fb4701e45369f4a1bd462c4edee"
format_stamp: "Formatted at 2026-09-26 01:14:17 on dist-test-slave-7b22"
server_key: "b8baf02a557f99e88425a39cfd2db6ca"
server_key_iv: "560c1718b25838345d10625ab7dd200b"
server_key_version: "encryptionkey@0"
I20260926 01:14:17.204753 22486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790384993574762-392-0/minicluster-data/master-0/wal/instance:
uuid: "1ff16fb4701e45369f4a1bd462c4edee"
format_stamp: "Formatted at 2026-09-26 01:14:17 on dist-test-slave-7b22"
server_key: "b8baf02a557f99e88425a39cfd2db6ca"
server_key_iv: "560c1718b25838345d10625ab7dd200b"
server_key_version: "encryptionkey@0"
I20260926 01:14:17.206152 22486 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 01:14:17.206986 22501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:14:17.207176 22486 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:14:17.207259 22486 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790384993574762-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790384993574762-392-0/minicluster-data/master-0/wal
uuid: "1ff16fb4701e45369f4a1bd462c4edee"
format_stamp: "Formatted at 2026-09-26 01:14:17 on dist-test-slave-7b22"
server_key: "b8baf02a557f99e88425a39cfd2db6ca"
server_key_iv: "560c1718b25838345d10625ab7dd200b"
server_key_version: "encryptionkey@0"
I20260926 01:14:17.207320 22486 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790384993574762-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790384993574762-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790384993574762-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:14:17.210649 22486 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:14:17.212805 22486 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:14:17.212920 22486 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:14:17.216763 22486 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42999
I20260926 01:14:17.216801 22553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42999 every 8 connection(s)
I20260926 01:14:17.217144 22486 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
I20260926 01:14:17.218505 22554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:14:17.220841 22554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ff16fb4701e45369f4a1bd462c4edee: Bootstrap starting.
I20260926 01:14:17.221400 22554 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ff16fb4701e45369f4a1bd462c4edee: Neither blocks nor log segments found. Creating new log.
I20260926 01:14:17.221657 22554 log.cc:824] T 00000000000000000000000000000000 P 1ff16fb4701e45369f4a1bd462c4edee: Log is configured to *not* fsync() on all Append() calls
I20260926 01:14:17.222268 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 22486
I20260926 01:14:17.222383 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790384993574762-392-0/minicluster-data/master-0/wal/instance
I20260926 01:14:17.222544 392 external_mini_cluster.cc:1428] Setting key 9290da007f55b3c2ae0f89b6d7079ce0
I20260926 01:14:17.222543 22554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ff16fb4701e45369f4a1bd462c4edee: No bootstrap required, opened a new log
I20260926 01:14:17.224715 22554 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ff16fb4701e45369f4a1bd462c4edee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ff16fb4701e45369f4a1bd462c4edee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42999 } }
I20260926 01:14:17.224915 22554 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1ff16fb4701e45369f4a1bd462c4edee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:14:17.224977 22554 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ff16fb4701e45369f4a1bd462c4edee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ff16fb4701e45369f4a1bd462c4edee, State: Initialized, Role: FOLLOWER
I20260926 01:14:17.225113 22554 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ff16fb4701e45369f4a1bd462c4edee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ff16fb4701e45369f4a1bd462c4edee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42999 } }
I20260926 01:14:17.225217 22554 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1ff16fb4701e45369f4a1bd462c4edee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:14:17.225243 22558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:14:17.223120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57489 (local address 127.0.98.62:42999)
0926 01:14:17.223305 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 01:14:17.223313 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 01:14:17.223341 (+ 28us) server_negotiation.cc:407] Connection header received
0926 01:14:17.223841 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:14:17.223848 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:14:17.223875 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:14:17.224002 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:14:17.224632 (+ 630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:14:17.224635 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:14:17.224636 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:14:17.224639 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:14:17.224681 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:14:17.224692 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:14:17.224745 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 01:14:17.224826 (+ 81us) server_negotiation.cc:299] Negotiation successful
0926 01:14:17.224863 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":84,"threads_started":1}
I20260926 01:14:17.225260 22554 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ff16fb4701e45369f4a1bd462c4edee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:14:17.225351 22554 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ff16fb4701e45369f4a1bd462c4edee [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:14:17.226128 22554 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ff16fb4701e45369f4a1bd462c4edee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ff16fb4701e45369f4a1bd462c4edee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42999 } }
I20260926 01:14:17.226248 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 01:14:17.226241 22554 leader_election.cc:302] T 00000000000000000000000000000000 P 1ff16fb4701e45369f4a1bd462c4edee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1ff16fb4701e45369f4a1bd462c4edee; no voters:
I20260926 01:14:17.226423 22554 leader_election.cc:288] T 00000000000000000000000000000000 P 1ff16fb4701e45369f4a1bd462c4edee [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:14:17.226456 22559 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ff16fb4701e45369f4a1bd462c4edee [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:14:17.226621 22559 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1ff16fb4701e45369f4a1bd462c4edee [term 1 LEADER]: Becoming Leader. State: Replica: 1ff16fb4701e45369f4a1bd462c4edee, State: Running, Role: LEADER
I20260926 01:14:17.226653 22554 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ff16fb4701e45369f4a1bd462c4edee [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:14:17.227959 22559 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ff16fb4701e45369f4a1bd462c4edee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ff16fb4701e45369f4a1bd462c4edee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42999 } }
I20260926 01:14:17.228552 22561 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ff16fb4701e45369f4a1bd462c4edee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1ff16fb4701e45369f4a1bd462c4edee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ff16fb4701e45369f4a1bd462c4edee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42999 } } }
I20260926 01:14:17.228644 22561 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ff16fb4701e45369f4a1bd462c4edee [sys.catalog]: This master's current role is: LEADER
I20260926 01:14:17.228868 22560 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ff16fb4701e45369f4a1bd462c4edee [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ff16fb4701e45369f4a1bd462c4edee. Latest consensus state: current_term: 1 leader_uuid: "1ff16fb4701e45369f4a1bd462c4edee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ff16fb4701e45369f4a1bd462c4edee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42999 } } }
I20260926 01:14:17.228928 22560 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ff16fb4701e45369f4a1bd462c4edee [sys.catalog]: This master's current role is: LEADER
I20260926 01:14:17.228981 22558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:14:17.227933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49721 (local address 127.0.98.62:42999)
0926 01:14:17.227966 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0926 01:14:17.227968 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:14:17.227975 (+ 7us) server_negotiation.cc:407] Connection header received
0926 01:14:17.227995 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:14:17.227996 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:14:17.228000 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:14:17.228046 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:14:17.228651 (+ 605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:14:17.228652 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:14:17.228652 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:14:17.228657 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:14:17.228670 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:14:17.228675 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:14:17.228729 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 01:14:17.228884 (+ 155us) server_negotiation.cc:299] Negotiation successful
0926 01:14:17.228896 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
W20260926 01:14:17.229132 22578 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1ff16fb4701e45369f4a1bd462c4edee: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 01:14:17.229187 22578 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 01:14:17.229301 22579 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:14:17.229517 22579 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:14:17.231029 22579 catalog_manager.cc:1347] Generated new cluster ID: 83a39f79acf94cca979e8ba698b7aef0
I20260926 01:14:17.231081 22579 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 01:14:17.249589 22579 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:14:17.249787 22579 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:14:17.260375 22579 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1ff16fb4701e45369f4a1bd462c4edee: Generated new TSK 0
I20260926 01:14:17.260601 22579 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 01:14:17.263604 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 22486
2026-09-26T01:14:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1240 ms)
[----------] 18 tests from AuthTokenIssuingTest (22291 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384993574762-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 01:14:17 dist-test-slave-7b22 krb5kdc[22585](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 01:14:17 dist-test-slave-7b22 krb5kdc[22585](info): set up 2 sockets
Sep 26 01:14:17 dist-test-slave-7b22 krb5kdc[22585](info): commencing operation
krb5kdc: starting...
W20260926 01:14:18.231065 392 mini_kdc.cc:121] Time spent starting KDC: real 0.950s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384993574762-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384993574762-392-0/krb5kdc/test-user.keytab.
Sep 26 01:14:18 dist-test-slave-7b22 krb5kdc[22585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385258, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T01:14:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:14:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384993574762-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384993574762-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 01:14:18.353555 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384993574762-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384993574762-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384993574762-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36503
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384993574762-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36503,127.0.98.61:34597,127.0.98.60:41699
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:14:18.436514 22598 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 01:14:18.436759 22598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:14:18.436776 22598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:14:18.436797 22598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:14:18.437989 22598 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:14:18.438028 22598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:14:18.438041 22598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:14:18.438053 22598 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:14:18.438064 22598 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:14:18.439575 22598 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384993574762-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384993574762-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36503,127.0.98.61:34597,127.0.98.60:41699
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384993574762-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36503
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22598
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384993574762-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:14:18.439798 22598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:14:18.439993 22598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:14:18.445008 22603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:14:18.445081 22604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:14:18.445261 22598 server_base.cc:1034] running on GCE node
W20260926 01:14:18.445083 22606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:14:18.445588 22598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:14:18.445933 22598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:14:18.447098 22598 hybrid_clock.cc:648] HybridClock initialized: now 1790385258447083 us; error 28 us; skew 500 ppm
Sep 26 01:14:18 dist-test-slave-7b22 krb5kdc[22585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385258, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:14:18.449812 22598 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 01:14:18.450305 22598 webserver.cc:460] Webserver started at http://127.0.98.62:42443/ using document root <none> and password file <none>
I20260926 01:14:18.450527 22598 fs_manager.cc:359] Metadata directory not provided
I20260926 01:14:18.450592 22598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:14:18.450703 22598 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:14:18.451810 22598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384993574762-392-0/minicluster-data/master-0/data/instance:
uuid: "fe88462cf9754df18cbc3e92e8ffda56"
format_stamp: "Formatted at 2026-09-26 01:14:18 on dist-test-slave-7b22"
server_key: "cf808365b68b3579be908bd47f981bf7"
server_key_iv: "37e0b486ea795024a2bac534674ad6eb"
server_key_version: "encryptionkey@0"
I20260926 01:14:18.452135 22598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384993574762-392-0/minicluster-data/master-0/wal/instance:
uuid: "fe88462cf9754df18cbc3e92e8ffda56"
format_stamp: "Formatted at 2026-09-26 01:14:18 on dist-test-slave-7b22"
server_key: "cf808365b68b3579be908bd47f981bf7"
server_key_iv: "37e0b486ea795024a2bac534674ad6eb"
server_key_version: "encryptionkey@0"
I20260926 01:14:18.453451 22598 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 01:14:18.454193 22613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:14:18.454340 22598 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 01:14:18.454427 22598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384993574762-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384993574762-392-0/minicluster-data/master-0/wal
uuid: "fe88462cf9754df18cbc3e92e8ffda56"
format_stamp: "Formatted at 2026-09-26 01:14:18 on dist-test-slave-7b22"
server_key: "cf808365b68b3579be908bd47f981bf7"
server_key_iv: "37e0b486ea795024a2bac534674ad6eb"
server_key_version: "encryptionkey@0"
I20260926 01:14:18.454490 22598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384993574762-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384993574762-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384993574762-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:14:18.465868 22598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:14:18.467689 22598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:14:18.467837 22598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:14:18.471573 22598 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36503
I20260926 01:14:18.471621 22665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36503 every 8 connection(s)
I20260926 01:14:18.471949 22598 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
I20260926 01:14:18.472870 22666 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:14:18.474541 22666 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36503 } has no permanent_uuid. Determining permanent_uuid...
I20260926 01:14:18.478701 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 22598
I20260926 01:14:18.478802 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384993574762-392-0/minicluster-data/master-0/wal/instance
I20260926 01:14:18.478999 392 external_mini_cluster.cc:1428] Setting key e5aaa94f9ca11f5394baa1fe55b231dd
Sep 26 01:14:18 dist-test-slave-7b22 krb5kdc[22585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385258, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:14:18.485301 22668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:14:18.475215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36503 (local address 127.0.0.1:39684)
0926 01:14:18.475424 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0926 01:14:18.475467 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 01:14:18.475949 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:14:18.476224 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:14:18.476227 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:14:18.476509 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:14:18.476668 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:14:18.476673 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:14:18.478067 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:14:18.478068 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:14:18.478957 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:14:18.478960 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:14:18.479480 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:14:18.479481 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:14:18.479544 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:14:18.480077 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:14:18.480086 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:14:18.481661 (+ 1575us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:14:18.483678 (+ 2017us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:14:18.483685 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:14:18.483686 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:14:18.483883 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:14:18.484163 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:14:18.484164 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:14:18.484164 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:14:18.484254 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:14:18.484711 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:14:18.484718 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:14:18.484781 (+ 63us) client_negotiation.cc:769] Sending connection context
0926 01:14:18.484867 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 01:14:18.485109 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"mutex_wait_us":100,"thread_start_us":74,"threads_started":1}
I20260926 01:14:18.485307 22669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:14:18.475379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39684 (local address 127.0.98.62:36503)
0926 01:14:18.475499 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0926 01:14:18.475503 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 01:14:18.475529 (+ 26us) server_negotiation.cc:407] Connection header received
0926 01:14:18.476080 (+ 551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:14:18.476082 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:14:18.476100 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:14:18.476177 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:14:18.476767 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:14:18.477961 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:14:18.479034 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:14:18.479387 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:14:18.479489 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:14:18.481740 (+ 2251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:14:18.481749 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:14:18.481749 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:14:18.481757 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:14:18.483587 (+ 1830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:14:18.483964 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:14:18.483967 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:14:18.483968 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:14:18.484073 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:14:18.484321 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:14:18.484322 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:14:18.484322 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:14:18.484632 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:14:18.484722 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 01:14:18.484913 (+ 191us) server_negotiation.cc:299] Negotiation successful
0926 01:14:18.485044 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":49,"threads_started":1}
I20260926 01:14:18.486326 22666 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34597 } has no permanent_uuid. Determining permanent_uuid...
I20260926 01:14:18.486694 22668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:14:18.486585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34597 (local address 127.0.0.1:49302)
0926 01:14:18.486629 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0926 01:14:18.486672 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260926 01:14:18.486831 22614 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34597: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
I20260926 01:14:18.487282 22668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:14:18.487192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34597 (local address 127.0.0.1:49318)
0926 01:14:18.487234 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0926 01:14:18.487262 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260926 01:14:18.487447 22666 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:34597: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34597: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384993574762-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384993574762-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260926 01:14:18.505362 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384993574762-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384993574762-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384993574762-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384993574762-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:34597
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384993574762-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36503,127.0.98.61:34597,127.0.98.60:41699
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 01:14:18.536677 22666 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34597 } attempt: 1
I20260926 01:14:18.537261 22668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:14:18.537093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34597 (local address 127.0.0.1:49324)
0926 01:14:18.537178 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0926 01:14:18.537227 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260926 01:14:18.537618 22668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:14:18.537530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34597 (local address 127.0.0.1:49334)
0926 01:14:18.537572 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0926 01:14:18.537599 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260926 01:14:18.537711 22666 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:34597: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34597: connect: Connection refused (error 111)
W20260926 01:14:18.592227 22673 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 01:14:18.592599 22673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:14:18.592658 22673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:14:18.592712 22673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:14:18.594820 22673 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:14:18.594890 22673 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:14:18.594910 22673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:14:18.594926 22673 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:14:18.594952 22673 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:14:18.596707 22673 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384993574762-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384993574762-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36503,127.0.98.61:34597,127.0.98.60:41699
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384993574762-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:34597
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384993574762-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22673
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384993574762-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:14:18.596979 22673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:14:18.597200 22673 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:14:18.602051 22678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:14:18.602061 22681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:14:18.602159 22679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:14:18.602535 22673 server_base.cc:1034] running on GCE node
I20260926 01:14:18.602805 22673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:14:18.603183 22673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:14:18.604353 22673 hybrid_clock.cc:648] HybridClock initialized: now 1790385258604333 us; error 36 us; skew 500 ppm
I20260926 01:14:18.605893 22666 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34597 } attempt: 2
I20260926 01:14:18.606271 22668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:14:18.606154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34597 (local address 127.0.0.1:49344)
0926 01:14:18.606206 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0926 01:14:18.606237 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260926 01:14:18.606554 22668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:14:18.606479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34597 (local address 127.0.0.1:49346)
0926 01:14:18.606511 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0926 01:14:18.606533 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260926 01:14:18.606669 22666 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:34597: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34597: connect: Connection refused (error 111)
Sep 26 01:14:18 dist-test-slave-7b22 krb5kdc[22585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385258, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:14:18.612754 22673 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260926 01:14:18.613220 22673 webserver.cc:460] Webserver started at http://127.0.98.61:40703/ using document root <none> and password file <none>
I20260926 01:14:18.613449 22673 fs_manager.cc:359] Metadata directory not provided
I20260926 01:14:18.613499 22673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:14:18.613656 22673 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:14:18.614804 22673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384993574762-392-0/minicluster-data/master-1/data/instance:
uuid: "29c43aba55024b2997ea2cb1686a2250"
format_stamp: "Formatted at 2026-09-26 01:14:18 on dist-test-slave-7b22"
server_key: "b8d1c59c934b736f84eaa351d3c3d39b"
server_key_iv: "fa31e9700bfd6ac24802486767337e01"
server_key_version: "encryptionkey@0"
I20260926 01:14:18.615152 22673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384993574762-392-0/minicluster-data/master-1/wal/instance:
uuid: "29c43aba55024b2997ea2cb1686a2250"
format_stamp: "Formatted at 2026-09-26 01:14:18 on dist-test-slave-7b22"
server_key: "b8d1c59c934b736f84eaa351d3c3d39b"
server_key_iv: "fa31e9700bfd6ac24802486767337e01"
server_key_version: "encryptionkey@0"
I20260926 01:14:18.616457 22673 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 01:14:18.617182 22688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:14:18.617367 22673 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 01:14:18.617462 22673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384993574762-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384993574762-392-0/minicluster-data/master-1/wal
uuid: "29c43aba55024b2997ea2cb1686a2250"
format_stamp: "Formatted at 2026-09-26 01:14:18 on dist-test-slave-7b22"
server_key: "b8d1c59c934b736f84eaa351d3c3d39b"
server_key_iv: "fa31e9700bfd6ac24802486767337e01"
server_key_version: "encryptionkey@0"
I20260926 01:14:18.617540 22673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384993574762-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384993574762-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384993574762-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:14:18.643867 22673 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:14:18.646015 22673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:14:18.646163 22673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:14:18.649926 22673 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:34597
I20260926 01:14:18.649984 22740 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:34597 every 8 connection(s)
I20260926 01:14:18.650306 22673 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384993574762-392-0/minicluster-data/master-1/data/info.pb
I20260926 01:14:18.650614 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 22673
I20260926 01:14:18.650710 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384993574762-392-0/minicluster-data/master-1/wal/instance
I20260926 01:14:18.650889 392 external_mini_cluster.cc:1428] Setting key 92fbefb6b9615945aec0897bf9e9f9b1
I20260926 01:14:18.651379 22741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:14:18.652973 22741 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36503 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 26 01:14:18 dist-test-slave-7b22 krb5kdc[22585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385258, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:14:18.664345 22669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:14:18.653786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39700 (local address 127.0.98.62:36503)
0926 01:14:18.653962 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 01:14:18.653964 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:14:18.654179 (+ 215us) server_negotiation.cc:407] Connection header received
0926 01:14:18.654741 (+ 562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:14:18.654742 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:14:18.654756 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:14:18.654811 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:14:18.655702 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:14:18.656213 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:14:18.657733 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:14:18.658020 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:14:18.658094 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:14:18.661585 (+ 3491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:14:18.661586 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:14:18.661587 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:14:18.661595 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:14:18.662984 (+ 1389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:14:18.663421 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:14:18.663422 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:14:18.663423 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:14:18.663487 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:14:18.663706 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:14:18.663707 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:14:18.663707 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:14:18.663840 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:14:18.663900 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 01:14:18.664150 (+ 250us) server_negotiation.cc:299] Negotiation successful
0926 01:14:18.664241 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145}
I20260926 01:14:18.664499 22744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:14:18.653772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36503 (local address 127.0.0.1:39700)
0926 01:14:18.654043 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0926 01:14:18.654090 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 01:14:18.654639 (+ 549us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:14:18.654927 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:14:18.654932 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:14:18.655281 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:14:18.655587 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:14:18.655594 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:14:18.656309 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:14:18.656310 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:14:18.657626 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:14:18.657629 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:14:18.658115 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:14:18.658116 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:14:18.658273 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:14:18.658703 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:14:18.658711 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:14:18.661461 (+ 2750us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:14:18.663078 (+ 1617us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:14:18.663086 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:14:18.663088 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:14:18.663322 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:14:18.663555 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:14:18.663556 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:14:18.663556 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:14:18.663654 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:14:18.663897 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:14:18.663902 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:14:18.664028 (+ 126us) client_negotiation.cc:769] Sending connection context
0926 01:14:18.664114 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 01:14:18.664258 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":108,"threads_started":1}
I20260926 01:14:18.665134 22741 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34597 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 01:14:18 dist-test-slave-7b22 krb5kdc[22585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385258, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260926 01:14:18.673877 22744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:14:18.665388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34597 (local address 127.0.0.1:49362)
0926 01:14:18.665438 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0926 01:14:18.665450 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 01:14:18.665494 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:14:18.665718 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:14:18.665719 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:14:18.665890 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:14:18.665965 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:14:18.665967 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:14:18.666808 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:14:18.666809 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:14:18.667617 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:14:18.667620 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:14:18.668097 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:14:18.668098 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:14:18.668174 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:14:18.668593 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:14:18.668597 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:14:18.671001 (+ 2404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:14:18.672766 (+ 1765us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:14:18.672767 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:14:18.672767 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:14:18.673027 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:14:18.673247 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:14:18.673248 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:14:18.673248 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:14:18.673305 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:14:18.673581 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:14:18.673582 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:14:18.673627 (+ 45us) client_negotiation.cc:769] Sending connection context
0926 01:14:18.673687 (+ 60us) client_negotiation.cc:240] Negotiation successful
0926 01:14:18.673763 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260926 01:14:18.674078 22746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:14:18.665446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49362 (local address 127.0.98.61:34597)
0926 01:14:18.665592 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0926 01:14:18.665597 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 01:14:18.665615 (+ 18us) server_negotiation.cc:407] Connection header received
0926 01:14:18.665633 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:14:18.665634 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:14:18.665644 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:14:18.665702 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:14:18.666036 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:14:18.666729 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:14:18.667707 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:14:18.668024 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:14:18.668081 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:14:18.671102 (+ 3021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:14:18.671113 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:14:18.671114 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:14:18.671122 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:14:18.672678 (+ 1556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:14:18.673126 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:14:18.673129 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:14:18.673130 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:14:18.673182 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:14:18.673362 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:14:18.673363 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:14:18.673363 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:14:18.673521 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:14:18.673580 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 01:14:18.673840 (+ 260us) server_negotiation.cc:299] Negotiation successful
0926 01:14:18.673977 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":83,"threads_started":1}
I20260926 01:14:18.674659 22741 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41699 } has no permanent_uuid. Determining permanent_uuid...
I20260926 01:14:18.675081 22744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:14:18.674884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41699 (local address 127.0.0.1:49178)
0926 01:14:18.674952 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0926 01:14:18.675039 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260926 01:14:18.675233 22690 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41699: connect: Connection refused (error 111)
I20260926 01:14:18.675777 22744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:14:18.675674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41699 (local address 127.0.0.1:49180)
0926 01:14:18.675730 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0926 01:14:18.675758 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260926 01:14:18.675961 22741 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41699: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41699: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384993574762-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384993574762-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260926 01:14:18.677093 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384993574762-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384993574762-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384993574762-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384993574762-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:41699
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384993574762-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36503,127.0.98.61:34597,127.0.98.60:41699
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 01:14:18.697875 22666 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34597 } attempt: 3
Sep 26 01:14:18 dist-test-slave-7b22 krb5kdc[22585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385258, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260926 01:14:18.705657 22668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:14:18.698265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34597 (local address 127.0.0.1:49376)
0926 01:14:18.698323 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0926 01:14:18.698336 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 01:14:18.698398 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:14:18.698564 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:14:18.698565 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:14:18.698711 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:14:18.698775 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:14:18.698776 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:14:18.699704 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:14:18.699705 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:14:18.700297 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:14:18.700301 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:14:18.700783 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:14:18.700784 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:14:18.700846 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:14:18.701496 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:14:18.701501 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:14:18.702827 (+ 1326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:14:18.704410 (+ 1583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:14:18.704411 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:14:18.704412 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:14:18.704648 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:14:18.704899 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:14:18.704901 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:14:18.704901 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:14:18.704969 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:14:18.705210 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:14:18.705211 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:14:18.705275 (+ 64us) client_negotiation.cc:769] Sending connection context
0926 01:14:18.705426 (+ 151us) client_negotiation.cc:240] Negotiation successful
0926 01:14:18.705550 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260926 01:14:18.705657 22746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:14:18.698351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49376 (local address 127.0.98.61:34597)
0926 01:14:18.698414 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 01:14:18.698417 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:14:18.698427 (+ 10us) server_negotiation.cc:407] Connection header received
0926 01:14:18.698457 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:14:18.698458 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:14:18.698471 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:14:18.698545 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:14:18.698841 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:14:18.699605 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:14:18.700388 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:14:18.700705 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:14:18.700775 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:14:18.702992 (+ 2217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:14:18.702994 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:14:18.702994 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:14:18.703001 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:14:18.704324 (+ 1323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:14:18.704746 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:14:18.704748 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:14:18.704748 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:14:18.704822 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:14:18.705028 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:14:18.705028 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:14:18.705029 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:14:18.705152 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:14:18.705209 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 01:14:18.705458 (+ 249us) server_negotiation.cc:299] Negotiation successful
0926 01:14:18.705552 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 01:14:18.706223 22666 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41699 } has no permanent_uuid. Determining permanent_uuid...
I20260926 01:14:18.706609 22668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:14:18.706484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41699 (local address 127.0.0.1:49182)
0926 01:14:18.706549 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0926 01:14:18.706583 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260926 01:14:18.706939 22668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:14:18.706845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41699 (local address 127.0.0.1:49188)
0926 01:14:18.706893 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0926 01:14:18.706916 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260926 01:14:18.707022 22666 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41699: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41699: connect: Connection refused (error 111)
I20260926 01:14:18.725193 22741 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41699 } attempt: 1
I20260926 01:14:18.725674 22744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:14:18.725528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41699 (local address 127.0.0.1:49194)
0926 01:14:18.725587 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0926 01:14:18.725635 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260926 01:14:18.726071 22744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:14:18.725987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41699 (local address 127.0.0.1:49200)
0926 01:14:18.726025 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0926 01:14:18.726051 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260926 01:14:18.726166 22741 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41699: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41699: connect: Connection refused (error 111)
I20260926 01:14:18.738181 22666 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41699 } attempt: 1
I20260926 01:14:18.738863 22668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:14:18.738692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41699 (local address 127.0.0.1:49214)
0926 01:14:18.738779 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0926 01:14:18.738825 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260926 01:14:18.739238 22668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:14:18.739149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41699 (local address 127.0.0.1:49224)
0926 01:14:18.739194 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0926 01:14:18.739218 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260926 01:14:18.739326 22666 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41699: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41699: connect: Connection refused (error 111)
W20260926 01:14:18.761857 22748 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 01:14:18.762111 22748 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:14:18.762135 22748 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:14:18.762163 22748 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:14:18.763370 22748 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:14:18.763415 22748 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:14:18.763433 22748 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:14:18.763449 22748 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:14:18.763464 22748 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:14:18.765020 22748 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384993574762-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384993574762-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36503,127.0.98.61:34597,127.0.98.60:41699
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384993574762-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:41699
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384993574762-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22748
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384993574762-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:14:18.765277 22748 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:14:18.765499 22748 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:14:18.770349 22753 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:14:18.770284 22754 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:14:18.770362 22756 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:14:18.770473 22748 server_base.cc:1034] running on GCE node
I20260926 01:14:18.770757 22748 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:14:18.771061 22748 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:14:18.772255 22748 hybrid_clock.cc:648] HybridClock initialized: now 1790385258772233 us; error 35 us; skew 500 ppm
Sep 26 01:14:18 dist-test-slave-7b22 krb5kdc[22585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385258, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:14:18.774739 22748 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260926 01:14:18.775163 22748 webserver.cc:460] Webserver started at http://127.0.98.60:45835/ using document root <none> and password file <none>
I20260926 01:14:18.775357 22748 fs_manager.cc:359] Metadata directory not provided
I20260926 01:14:18.775395 22748 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:14:18.775489 22748 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:14:18.776546 22748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384993574762-392-0/minicluster-data/master-2/data/instance:
uuid: "fad13ee3df594713bf1a627c9a4a8175"
format_stamp: "Formatted at 2026-09-26 01:14:18 on dist-test-slave-7b22"
server_key: "c3109f4d29ae57445ec5c112ebb9beb1"
server_key_iv: "e3aa669eca9e2d7c267b9fbb576a41f1"
server_key_version: "encryptionkey@0"
I20260926 01:14:18.776837 22748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384993574762-392-0/minicluster-data/master-2/wal/instance:
uuid: "fad13ee3df594713bf1a627c9a4a8175"
format_stamp: "Formatted at 2026-09-26 01:14:18 on dist-test-slave-7b22"
server_key: "c3109f4d29ae57445ec5c112ebb9beb1"
server_key_iv: "e3aa669eca9e2d7c267b9fbb576a41f1"
server_key_version: "encryptionkey@0"
I20260926 01:14:18.778110 22748 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 01:14:18.778784 22763 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:14:18.778949 22748 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 01:14:18.779027 22748 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384993574762-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384993574762-392-0/minicluster-data/master-2/wal
uuid: "fad13ee3df594713bf1a627c9a4a8175"
format_stamp: "Formatted at 2026-09-26 01:14:18 on dist-test-slave-7b22"
server_key: "c3109f4d29ae57445ec5c112ebb9beb1"
server_key_iv: "e3aa669eca9e2d7c267b9fbb576a41f1"
server_key_version: "encryptionkey@0"
I20260926 01:14:18.779081 22748 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384993574762-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384993574762-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384993574762-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:14:18.793728 22748 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:14:18.794324 22741 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41699 } attempt: 2
I20260926 01:14:18.794806 22744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:14:18.794648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41699 (local address 127.0.0.1:49236)
0926 01:14:18.794731 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0926 01:14:18.794772 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260926 01:14:18.795250 22744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:14:18.795095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41699 (local address 127.0.0.1:49250)
0926 01:14:18.795137 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0926 01:14:18.795224 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260926 01:14:18.795394 22741 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41699: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41699: connect: Connection refused (error 111)
I20260926 01:14:18.796268 22748 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:14:18.796378 22748 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:14:18.800148 22748 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:41699
I20260926 01:14:18.800182 22815 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:41699 every 8 connection(s)
I20260926 01:14:18.800509 22748 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384993574762-392-0/minicluster-data/master-2/data/info.pb
I20260926 01:14:18.801514 22816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:14:18.801931 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 22748
I20260926 01:14:18.802028 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384993574762-392-0/minicluster-data/master-2/wal/instance
I20260926 01:14:18.802188 392 external_mini_cluster.cc:1428] Setting key e93ab56703847d6e74efeb38c193949b
I20260926 01:14:18.803435 22816 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36503 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 01:14:18 dist-test-slave-7b22 krb5kdc[22585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385258, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 26 01:14:18 dist-test-slave-7b22 krb5kdc[22585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385258, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260926 01:14:18.812119 22669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:14:18.804058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39704 (local address 127.0.98.62:36503)
0926 01:14:18.804111 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 01:14:18.804112 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 01:14:18.804193 (+ 81us) server_negotiation.cc:407] Connection header received
0926 01:14:18.804238 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:14:18.804238 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:14:18.804249 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:14:18.804304 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:14:18.804753 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:14:18.805470 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:14:18.806683 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:14:18.807002 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:14:18.807062 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:14:18.809321 (+ 2259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:14:18.809323 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:14:18.809323 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:14:18.809331 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:14:18.810642 (+ 1311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:14:18.811104 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:14:18.811105 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:14:18.811106 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:14:18.811171 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:14:18.811409 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:14:18.811410 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:14:18.811410 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:14:18.811562 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:14:18.811635 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 01:14:18.811936 (+ 301us) server_negotiation.cc:299] Negotiation successful
0926 01:14:18.812021 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260926 01:14:18.812270 22820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:14:18.803948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36503 (local address 127.0.0.1:39704)
0926 01:14:18.804094 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0926 01:14:18.804135 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 01:14:18.804194 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:14:18.804321 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:14:18.804323 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:14:18.804574 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:14:18.804674 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:14:18.804678 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:14:18.805567 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:14:18.805567 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:14:18.806600 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:14:18.806602 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:14:18.807094 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:14:18.807095 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:14:18.807164 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:14:18.807664 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:14:18.807673 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:14:18.809229 (+ 1556us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:14:18.810709 (+ 1480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:14:18.810716 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:14:18.810719 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:14:18.811013 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:14:18.811237 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:14:18.811237 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:14:18.811237 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:14:18.811340 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:14:18.811624 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:14:18.811627 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:14:18.811784 (+ 157us) client_negotiation.cc:769] Sending connection context
0926 01:14:18.811898 (+ 114us) client_negotiation.cc:240] Negotiation successful
0926 01:14:18.812045 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":72,"threads_started":1}
I20260926 01:14:18.812906 22816 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34597 } has no permanent_uuid. Determining permanent_uuid...
I20260926 01:14:18.813032 22819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:14:18.802710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49262 (local address 127.0.98.60:41699)
0926 01:14:18.802922 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0926 01:14:18.802931 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 01:14:18.802955 (+ 24us) server_negotiation.cc:407] Connection header received
0926 01:14:18.803360 (+ 405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:14:18.803366 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:14:18.803411 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:14:18.803545 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:14:18.804204 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:14:18.805423 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:14:18.806506 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:14:18.806912 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:14:18.807034 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:14:18.809784 (+ 2750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:14:18.809792 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:14:18.809793 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:14:18.809799 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:14:18.811628 (+ 1829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:14:18.811982 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:14:18.811985 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:14:18.811985 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:14:18.812046 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:14:18.812277 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:14:18.812278 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:14:18.812278 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:14:18.812627 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:14:18.812761 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0926 01:14:18.812873 (+ 112us) server_negotiation.cc:299] Negotiation successful
0926 01:14:18.812948 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":93,"threads_started":1}
I20260926 01:14:18.814448 22666 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41699 } attempt: 2
Sep 26 01:14:18 dist-test-slave-7b22 krb5kdc[22585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385258, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 26 01:14:18 dist-test-slave-7b22 krb5kdc[22585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385258, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 26 01:14:18 dist-test-slave-7b22 krb5kdc[22585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385258, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:14:18.821094 22668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:14:18.814688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41699 (local address 127.0.0.1:49264)
0926 01:14:18.814723 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0926 01:14:18.814732 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 01:14:18.814765 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:14:18.814886 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:14:18.814887 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:14:18.815033 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:14:18.815068 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:14:18.815070 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:14:18.815741 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:14:18.815743 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:14:18.816411 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:14:18.816413 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:14:18.816732 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:14:18.816733 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:14:18.816782 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:14:18.817112 (+ 330us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:14:18.817116 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:14:18.818249 (+ 1133us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:14:18.820048 (+ 1799us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:14:18.820049 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:14:18.820049 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:14:18.820282 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:14:18.820501 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:14:18.820502 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:14:18.820502 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:14:18.820560 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:14:18.820814 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:14:18.820815 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:14:18.820866 (+ 51us) client_negotiation.cc:769] Sending connection context
0926 01:14:18.820920 (+ 54us) client_negotiation.cc:240] Negotiation successful
0926 01:14:18.820996 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19}
I20260926 01:14:18.821138 22819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:14:18.814743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49264 (local address 127.0.98.60:41699)
0926 01:14:18.814781 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 01:14:18.814783 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:14:18.814789 (+ 6us) server_negotiation.cc:407] Connection header received
0926 01:14:18.814806 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:14:18.814806 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:14:18.814815 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:14:18.815127 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:14:18.815152 (+ 25us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:14:18.815678 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:14:18.816476 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:14:18.816677 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:14:18.816726 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:14:18.818333 (+ 1607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:14:18.818334 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:14:18.818335 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:14:18.818342 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:14:18.819767 (+ 1425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:14:18.820389 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:14:18.820390 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:14:18.820391 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:14:18.820450 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:14:18.820611 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:14:18.820612 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:14:18.820612 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:14:18.820726 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:14:18.820816 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 01:14:18.820958 (+ 142us) server_negotiation.cc:299] Negotiation successful
0926 01:14:18.821027 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 01:14:18.821931 22669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:14:18.813863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39712 (local address 127.0.98.62:36503)
0926 01:14:18.813896 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0926 01:14:18.813898 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:14:18.813905 (+ 7us) server_negotiation.cc:407] Connection header received
0926 01:14:18.813955 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:14:18.813956 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:14:18.813963 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:14:18.814003 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:14:18.814538 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:14:18.815055 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:14:18.815837 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:14:18.816109 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:14:18.816184 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:14:18.818906 (+ 2722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:14:18.818908 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:14:18.818908 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:14:18.818914 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:14:18.820394 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:14:18.821188 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:14:18.821189 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:14:18.821190 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:14:18.821234 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:14:18.821418 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:14:18.821419 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:14:18.821420 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:14:18.821556 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:14:18.821686 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0926 01:14:18.821759 (+ 73us) server_negotiation.cc:299] Negotiation successful
0926 01:14:18.821855 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260926 01:14:18.822325 22820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:14:18.813171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34597 (local address 127.0.0.1:49378)
0926 01:14:18.813212 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0926 01:14:18.813222 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 01:14:18.813269 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:14:18.813419 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:14:18.813420 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:14:18.813556 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:14:18.813605 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:14:18.813606 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:14:18.814489 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:14:18.814490 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:14:18.815328 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:14:18.815329 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:14:18.815853 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:14:18.815854 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:14:18.815917 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:14:18.816399 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:14:18.816403 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:14:18.819103 (+ 2700us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:14:18.820773 (+ 1670us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:14:18.820773 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:14:18.820774 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:14:18.820963 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:14:18.821660 (+ 697us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:14:18.821661 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:14:18.821661 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:14:18.821723 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:14:18.822054 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:14:18.822055 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:14:18.822095 (+ 40us) client_negotiation.cc:769] Sending connection context
0926 01:14:18.822159 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 01:14:18.822245 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260926 01:14:18.822346 22746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:14:18.813266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49378 (local address 127.0.98.61:34597)
0926 01:14:18.813322 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 01:14:18.813324 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:14:18.813330 (+ 6us) server_negotiation.cc:407] Connection header received
0926 01:14:18.813350 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:14:18.813351 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:14:18.813360 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:14:18.813402 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:14:18.813669 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:14:18.814397 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:14:18.815405 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:14:18.815731 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:14:18.815796 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:14:18.819178 (+ 3382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:14:18.819180 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:14:18.819180 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:14:18.819189 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:14:18.820693 (+ 1504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:14:18.821041 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:14:18.821042 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:14:18.821042 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:14:18.821100 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:14:18.821798 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:14:18.821799 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:14:18.821799 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:14:18.821962 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:14:18.822006 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0926 01:14:18.822204 (+ 198us) server_negotiation.cc:299] Negotiation successful
0926 01:14:18.822274 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 01:14:18.822716 22816 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41699 } has no permanent_uuid. Determining permanent_uuid...
I20260926 01:14:18.823618 22666 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe88462cf9754df18cbc3e92e8ffda56: Bootstrap starting.
I20260926 01:14:18.824265 22666 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fe88462cf9754df18cbc3e92e8ffda56: Neither blocks nor log segments found. Creating new log.
I20260926 01:14:18.824576 22666 log.cc:824] T 00000000000000000000000000000000 P fe88462cf9754df18cbc3e92e8ffda56: Log is configured to *not* fsync() on all Append() calls
I20260926 01:14:18.825474 22666 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe88462cf9754df18cbc3e92e8ffda56: No bootstrap required, opened a new log
Sep 26 01:14:18 dist-test-slave-7b22 krb5kdc[22585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385258, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260926 01:14:18.828027 22666 raft_consensus.cc:348] T 00000000000000000000000000000000 P fe88462cf9754df18cbc3e92e8ffda56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe88462cf9754df18cbc3e92e8ffda56" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36503 } } peers { permanent_uuid: "29c43aba55024b2997ea2cb1686a2250" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34597 } } peers { permanent_uuid: "fad13ee3df594713bf1a627c9a4a8175" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41699 } }
I20260926 01:14:18.828182 22666 raft_consensus.cc:729] T 00000000000000000000000000000000 P fe88462cf9754df18cbc3e92e8ffda56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe88462cf9754df18cbc3e92e8ffda56, State: Initialized, Role: FOLLOWER
I20260926 01:14:18.828289 22666 consensus_queue.cc:260] T 00000000000000000000000000000000 P fe88462cf9754df18cbc3e92e8ffda56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe88462cf9754df18cbc3e92e8ffda56" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36503 } } peers { permanent_uuid: "29c43aba55024b2997ea2cb1686a2250" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34597 } } peers { permanent_uuid: "fad13ee3df594713bf1a627c9a4a8175" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41699 } }
I20260926 01:14:18.828675 22666 sys_catalog.cc:564] T 00000000000000000000000000000000 P fe88462cf9754df18cbc3e92e8ffda56 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:14:18.828708 22822 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe88462cf9754df18cbc3e92e8ffda56 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe88462cf9754df18cbc3e92e8ffda56" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36503 } } peers { permanent_uuid: "29c43aba55024b2997ea2cb1686a2250" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34597 } } peers { permanent_uuid: "fad13ee3df594713bf1a627c9a4a8175" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41699 } } }
I20260926 01:14:18.828907 22822 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe88462cf9754df18cbc3e92e8ffda56 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 01:14:18.830785 22820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:14:18.823061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41699 (local address 127.0.0.1:49274)
0926 01:14:18.823152 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0926 01:14:18.823163 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 01:14:18.823218 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:14:18.823324 (+ 106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:14:18.823325 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:14:18.823478 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:14:18.823521 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:14:18.823522 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:14:18.824436 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:14:18.824437 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:14:18.825203 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:14:18.825207 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:14:18.825694 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:14:18.825696 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:14:18.825753 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:14:18.826229 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:14:18.826235 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:14:18.827984 (+ 1749us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:14:18.829689 (+ 1705us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:14:18.829690 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:14:18.829690 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:14:18.829903 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:14:18.830092 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:14:18.830093 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:14:18.830093 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:14:18.830150 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:14:18.830480 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:14:18.830481 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:14:18.830519 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 01:14:18.830594 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 01:14:18.830693 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78}
W20260926 01:14:18.830780 22833 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fe88462cf9754df18cbc3e92e8ffda56: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 01:14:18.830826 22833 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Sep 26 01:14:18 dist-test-slave-7b22 krb5kdc[22585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385258, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260926 01:14:18.831287 22819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:14:18.823110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49274 (local address 127.0.98.60:41699)
0926 01:14:18.823148 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 01:14:18.823149 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 01:14:18.823200 (+ 51us) server_negotiation.cc:407] Connection header received
0926 01:14:18.823269 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:14:18.823269 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:14:18.823276 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:14:18.823317 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:14:18.823678 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:14:18.824307 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:14:18.825294 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:14:18.825618 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:14:18.825680 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:14:18.828095 (+ 2415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:14:18.828097 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:14:18.828097 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:14:18.828105 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:14:18.829620 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:14:18.829987 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:14:18.829988 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:14:18.829988 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:14:18.830046 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:14:18.830208 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:14:18.830208 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:14:18.830209 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:14:18.830399 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:14:18.830460 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 01:14:18.831135 (+ 675us) server_negotiation.cc:299] Negotiation successful
0926 01:14:18.831212 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 01:14:18.833141 22816 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fad13ee3df594713bf1a627c9a4a8175: Bootstrap starting.
I20260926 01:14:18.834149 22816 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fad13ee3df594713bf1a627c9a4a8175: Neither blocks nor log segments found. Creating new log.
I20260926 01:14:18.831525 22598 master_runner.cc:186] Error getting master registration for 127.0.98.61:34597: OK, instance_id { permanent_uuid: "29c43aba55024b2997ea2cb1686a2250" instance_seqno: 1790385258648801 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 01:14:18.834441 22816 log.cc:824] T 00000000000000000000000000000000 P fad13ee3df594713bf1a627c9a4a8175: Log is configured to *not* fsync() on all Append() calls
I20260926 01:14:18.834509 22598 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 01:14:18.834514 22746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:14:18.825710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49382 (local address 127.0.98.61:34597)
0926 01:14:18.825748 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 01:14:18.825749 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 01:14:18.825935 (+ 186us) server_negotiation.cc:407] Connection header received
0926 01:14:18.826035 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:14:18.826036 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:14:18.826048 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:14:18.826104 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:14:18.826943 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:14:18.827420 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:14:18.828110 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:14:18.828400 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:14:18.828465 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:14:18.831338 (+ 2873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:14:18.831340 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:14:18.831340 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:14:18.831348 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:14:18.833116 (+ 1768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:14:18.833523 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:14:18.833524 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:14:18.833524 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:14:18.833583 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:14:18.833779 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:14:18.833780 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:14:18.833781 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:14:18.833972 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:14:18.834115 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0926 01:14:18.834335 (+ 220us) server_negotiation.cc:299] Negotiation successful
0926 01:14:18.834419 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 01:14:18.835227 22816 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fad13ee3df594713bf1a627c9a4a8175: No bootstrap required, opened a new log
I20260926 01:14:18.836159 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 01:14:18.837235 22640 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "fe88462cf9754df18cbc3e92e8ffda56"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39712
I20260926 01:14:18.837406 22640 raft_consensus.cc:482] T 00000000000000000000000000000000 P fe88462cf9754df18cbc3e92e8ffda56 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260926 01:14:18.837316 22816 raft_consensus.cc:348] T 00000000000000000000000000000000 P fad13ee3df594713bf1a627c9a4a8175 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe88462cf9754df18cbc3e92e8ffda56" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36503 } } peers { permanent_uuid: "29c43aba55024b2997ea2cb1686a2250" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34597 } } peers { permanent_uuid: "fad13ee3df594713bf1a627c9a4a8175" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41699 } }
I20260926 01:14:18.837457 22640 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fe88462cf9754df18cbc3e92e8ffda56 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:14:18.837456 22816 raft_consensus.cc:729] T 00000000000000000000000000000000 P fad13ee3df594713bf1a627c9a4a8175 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fad13ee3df594713bf1a627c9a4a8175, State: Initialized, Role: FOLLOWER
I20260926 01:14:18.837530 22816 consensus_queue.cc:260] T 00000000000000000000000000000000 P fad13ee3df594713bf1a627c9a4a8175 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe88462cf9754df18cbc3e92e8ffda56" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36503 } } peers { permanent_uuid: "29c43aba55024b2997ea2cb1686a2250" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34597 } } peers { permanent_uuid: "fad13ee3df594713bf1a627c9a4a8175" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41699 } }
I20260926 01:14:18.837854 22836 sys_catalog.cc:455] T 00000000000000000000000000000000 P fad13ee3df594713bf1a627c9a4a8175 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe88462cf9754df18cbc3e92e8ffda56" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36503 } } peers { permanent_uuid: "29c43aba55024b2997ea2cb1686a2250" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34597 } } peers { permanent_uuid: "fad13ee3df594713bf1a627c9a4a8175" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41699 } } }
I20260926 01:14:18.837950 22816 sys_catalog.cc:564] T 00000000000000000000000000000000 P fad13ee3df594713bf1a627c9a4a8175 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:14:18.837965 22836 sys_catalog.cc:458] T 00000000000000000000000000000000 P fad13ee3df594713bf1a627c9a4a8175 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 01:14:18.838426 22640 raft_consensus.cc:504] T 00000000000000000000000000000000 P fe88462cf9754df18cbc3e92e8ffda56 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe88462cf9754df18cbc3e92e8ffda56" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36503 } } peers { permanent_uuid: "29c43aba55024b2997ea2cb1686a2250" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34597 } } peers { permanent_uuid: "fad13ee3df594713bf1a627c9a4a8175" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41699 } }
I20260926 01:14:18.838699 22640 leader_election.cc:288] T 00000000000000000000000000000000 P fe88462cf9754df18cbc3e92e8ffda56 [CANDIDATE]: Term 1 election: Requested vote from peers 29c43aba55024b2997ea2cb1686a2250 (127.0.98.61:34597), fad13ee3df594713bf1a627c9a4a8175 (127.0.98.60:41699)
I20260926 01:14:18.838936 22790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fe88462cf9754df18cbc3e92e8ffda56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "fad13ee3df594713bf1a627c9a4a8175"
I20260926 01:14:18.839033 22790 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fad13ee3df594713bf1a627c9a4a8175 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:14:18.839753 22790 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fad13ee3df594713bf1a627c9a4a8175 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe88462cf9754df18cbc3e92e8ffda56 in term 1.
I20260926 01:14:18.840054 22615 leader_election.cc:302] T 00000000000000000000000000000000 P fe88462cf9754df18cbc3e92e8ffda56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: fad13ee3df594713bf1a627c9a4a8175, fe88462cf9754df18cbc3e92e8ffda56; no voters:
I20260926 01:14:18.840180 22822 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fe88462cf9754df18cbc3e92e8ffda56 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:14:18.840322 22822 raft_consensus.cc:686] T 00000000000000000000000000000000 P fe88462cf9754df18cbc3e92e8ffda56 [term 1 LEADER]: Becoming Leader. State: Replica: fe88462cf9754df18cbc3e92e8ffda56, State: Running, Role: LEADER
I20260926 01:14:18.840411 22822 consensus_queue.cc:237] T 00000000000000000000000000000000 P fe88462cf9754df18cbc3e92e8ffda56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe88462cf9754df18cbc3e92e8ffda56" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36503 } } peers { permanent_uuid: "29c43aba55024b2997ea2cb1686a2250" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34597 } } peers { permanent_uuid: "fad13ee3df594713bf1a627c9a4a8175" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41699 } }
I20260926 01:14:18.840876 22822 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe88462cf9754df18cbc3e92e8ffda56 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fe88462cf9754df18cbc3e92e8ffda56. Latest consensus state: current_term: 1 leader_uuid: "fe88462cf9754df18cbc3e92e8ffda56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe88462cf9754df18cbc3e92e8ffda56" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36503 } } peers { permanent_uuid: "29c43aba55024b2997ea2cb1686a2250" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34597 } } peers { permanent_uuid: "fad13ee3df594713bf1a627c9a4a8175" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41699 } } }
I20260926 01:14:18.840956 22822 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe88462cf9754df18cbc3e92e8ffda56 [sys.catalog]: This master's current role is: LEADER
I20260926 01:14:18.841779 22851 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:14:18.842069 22851 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260926 01:14:18.845184 22857 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fad13ee3df594713bf1a627c9a4a8175: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 01:14:18.847196 22857 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 01:14:18.843863 22748 master_runner.cc:186] Error getting master registration for 127.0.98.61:34597: OK, instance_id { permanent_uuid: "29c43aba55024b2997ea2cb1686a2250" instance_seqno: 1790385258648801 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 01:14:18.846930 22790 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fad13ee3df594713bf1a627c9a4a8175 [term 1 FOLLOWER]: Refusing update from remote peer fe88462cf9754df18cbc3e92e8ffda56: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:14:18.847558 22748 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 01:14:18.847784 22848 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fe88462cf9754df18cbc3e92e8ffda56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fad13ee3df594713bf1a627c9a4a8175" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41699 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:14:18.849305 22746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:14:18.840003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49384 (local address 127.0.98.61:34597)
0926 01:14:18.840047 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 01:14:18.840049 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:14:18.840548 (+ 499us) server_negotiation.cc:407] Connection header received
0926 01:14:18.840642 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:14:18.840643 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:14:18.840655 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:14:18.840683 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:14:18.841532 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:14:18.842233 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:14:18.843584 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:14:18.843869 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:14:18.843911 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:14:18.846392 (+ 2481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:14:18.846394 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:14:18.846394 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:14:18.846402 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:14:18.847971 (+ 1569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:14:18.848462 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:14:18.848464 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:14:18.848464 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:14:18.848522 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:14:18.848717 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:14:18.848718 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:14:18.848718 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:14:18.848855 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:14:18.848922 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 01:14:18.849131 (+ 209us) server_negotiation.cc:299] Negotiation successful
0926 01:14:18.849215 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 01:14:18.849300 22836 sys_catalog.cc:455] T 00000000000000000000000000000000 P fad13ee3df594713bf1a627c9a4a8175 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fe88462cf9754df18cbc3e92e8ffda56. Latest consensus state: current_term: 1 leader_uuid: "fe88462cf9754df18cbc3e92e8ffda56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe88462cf9754df18cbc3e92e8ffda56" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36503 } } peers { permanent_uuid: "29c43aba55024b2997ea2cb1686a2250" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34597 } } peers { permanent_uuid: "fad13ee3df594713bf1a627c9a4a8175" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41699 } } }
I20260926 01:14:18.849399 22836 sys_catalog.cc:458] T 00000000000000000000000000000000 P fad13ee3df594713bf1a627c9a4a8175 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 01:14:18.856642 22669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:14:18.839963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39726 (local address 127.0.98.62:36503)
0926 01:14:18.840005 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 01:14:18.840007 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:14:18.849749 (+ 9742us) server_negotiation.cc:407] Connection header received
0926 01:14:18.849867 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:14:18.849868 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:14:18.849880 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:14:18.849928 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:14:18.850495 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:14:18.851216 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:14:18.852128 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:14:18.852408 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:14:18.852451 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:14:18.853856 (+ 1405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:14:18.853857 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:14:18.853857 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:14:18.853865 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:14:18.855351 (+ 1486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:14:18.855774 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:14:18.855776 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:14:18.855776 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:14:18.855844 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:14:18.856045 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:14:18.856046 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:14:18.856046 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:14:18.856175 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:14:18.856248 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 01:14:18.856455 (+ 207us) server_negotiation.cc:299] Negotiation successful
0926 01:14:18.856547 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 01:14:18.858563 22819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:14:18.840291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49290 (local address 127.0.98.60:41699)
0926 01:14:18.840326 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0926 01:14:18.840327 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 01:14:18.850003 (+ 9676us) server_negotiation.cc:407] Connection header received
0926 01:14:18.850122 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:14:18.850123 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:14:18.850135 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:14:18.850183 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:14:18.851161 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:14:18.851865 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:14:18.853181 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:14:18.853529 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:14:18.853573 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:14:18.855824 (+ 2251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:14:18.855839 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:14:18.855839 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:14:18.855846 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:14:18.857384 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:14:18.857733 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:14:18.857734 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:14:18.857734 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:14:18.857802 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:14:18.858031 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:14:18.858032 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:14:18.858032 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:14:18.858155 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:14:18.858247 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 01:14:18.858401 (+ 154us) server_negotiation.cc:299] Negotiation successful
0926 01:14:18.858470 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 01:14:18.859376 22848 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe88462cf9754df18cbc3e92e8ffda56 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fe88462cf9754df18cbc3e92e8ffda56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe88462cf9754df18cbc3e92e8ffda56" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36503 } } peers { permanent_uuid: "29c43aba55024b2997ea2cb1686a2250" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34597 } } peers { permanent_uuid: "fad13ee3df594713bf1a627c9a4a8175" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41699 } } }
I20260926 01:14:18.859469 22848 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe88462cf9754df18cbc3e92e8ffda56 [sys.catalog]: This master's current role is: LEADER
I20260926 01:14:18.859484 22836 sys_catalog.cc:455] T 00000000000000000000000000000000 P fad13ee3df594713bf1a627c9a4a8175 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fe88462cf9754df18cbc3e92e8ffda56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe88462cf9754df18cbc3e92e8ffda56" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36503 } } peers { permanent_uuid: "29c43aba55024b2997ea2cb1686a2250" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34597 } } peers { permanent_uuid: "fad13ee3df594713bf1a627c9a4a8175" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41699 } } }
I20260926 01:14:18.859575 22836 sys_catalog.cc:458] T 00000000000000000000000000000000 P fad13ee3df594713bf1a627c9a4a8175 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 01:14:18.859947 22860 mvcc.cc:205] Tried to move back new op lower bound from 7333418020220977152 to 7333418020211138560. Current Snapshot: MvccSnapshot[applied={T|T < 7333418020220977152}]
I20260926 01:14:18.860059 22858 mvcc.cc:205] Tried to move back new op lower bound from 7333418020220977152 to 7333418020211138560. Current Snapshot: MvccSnapshot[applied={T|T < 7333418020220977152}]
I20260926 01:14:18.860327 22851 catalog_manager.cc:1347] Generated new cluster ID: dba77eb4674b4ae5af696916f2e6fb93
I20260926 01:14:18.860365 22851 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 01:14:18.872103 22851 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:14:18.872936 22851 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:14:18.884861 22851 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fe88462cf9754df18cbc3e92e8ffda56: Generated new TSK 0
I20260926 01:14:18.885221 22851 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 01:14:18.886507 22741 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41699 } attempt: 3
Sep 26 01:14:18 dist-test-slave-7b22 krb5kdc[22585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385258, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260926 01:14:18.896344 22744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:14:18.886812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41699 (local address 127.0.0.1:49304)
0926 01:14:18.886912 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0926 01:14:18.886924 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 01:14:18.886965 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:14:18.887139 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:14:18.887140 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:14:18.887299 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:14:18.887342 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:14:18.887344 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:14:18.888250 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:14:18.888252 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:14:18.888750 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:14:18.888753 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:14:18.889369 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:14:18.889370 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:14:18.889427 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:14:18.889960 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:14:18.889965 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:14:18.891551 (+ 1586us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:14:18.894719 (+ 3168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:14:18.894720 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:14:18.894720 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:14:18.894940 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:14:18.895653 (+ 713us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:14:18.895654 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:14:18.895655 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:14:18.895719 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:14:18.896015 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:14:18.896016 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:14:18.896065 (+ 49us) client_negotiation.cc:769] Sending connection context
0926 01:14:18.896129 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 01:14:18.896242 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77}
I20260926 01:14:18.897092 22819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:14:18.886912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49304 (local address 127.0.98.60:41699)
0926 01:14:18.887040 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0926 01:14:18.887042 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:14:18.887048 (+ 6us) server_negotiation.cc:407] Connection header received
0926 01:14:18.887069 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:14:18.887069 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:14:18.887079 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:14:18.887122 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:14:18.887423 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:14:18.888173 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:14:18.888827 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:14:18.889124 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:14:18.889177 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:14:18.893042 (+ 3865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:14:18.893043 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:14:18.893043 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:14:18.893052 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:14:18.894636 (+ 1584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:14:18.895511 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:14:18.895512 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:14:18.895513 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:14:18.895575 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:14:18.895767 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:14:18.895768 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:14:18.895769 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:14:18.895925 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:14:18.895986 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 01:14:18.896921 (+ 935us) server_negotiation.cc:299] Negotiation successful
0926 01:14:18.897000 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260926 01:14:18.898846 22741 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29c43aba55024b2997ea2cb1686a2250: Bootstrap starting.
I20260926 01:14:18.899571 22741 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 29c43aba55024b2997ea2cb1686a2250: Neither blocks nor log segments found. Creating new log.
I20260926 01:14:18.899852 22741 log.cc:824] T 00000000000000000000000000000000 P 29c43aba55024b2997ea2cb1686a2250: Log is configured to *not* fsync() on all Append() calls
I20260926 01:14:18.900763 22741 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29c43aba55024b2997ea2cb1686a2250: No bootstrap required, opened a new log
I20260926 01:14:18.902777 22741 raft_consensus.cc:348] T 00000000000000000000000000000000 P 29c43aba55024b2997ea2cb1686a2250 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe88462cf9754df18cbc3e92e8ffda56" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36503 } } peers { permanent_uuid: "29c43aba55024b2997ea2cb1686a2250" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34597 } } peers { permanent_uuid: "fad13ee3df594713bf1a627c9a4a8175" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41699 } }
I20260926 01:14:18.902930 22741 raft_consensus.cc:729] T 00000000000000000000000000000000 P 29c43aba55024b2997ea2cb1686a2250 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29c43aba55024b2997ea2cb1686a2250, State: Initialized, Role: FOLLOWER
I20260926 01:14:18.903020 22741 consensus_queue.cc:260] T 00000000000000000000000000000000 P 29c43aba55024b2997ea2cb1686a2250 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe88462cf9754df18cbc3e92e8ffda56" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36503 } } peers { permanent_uuid: "29c43aba55024b2997ea2cb1686a2250" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34597 } } peers { permanent_uuid: "fad13ee3df594713bf1a627c9a4a8175" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41699 } }
I20260926 01:14:18.903359 22741 sys_catalog.cc:564] T 00000000000000000000000000000000 P 29c43aba55024b2997ea2cb1686a2250 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:14:18.903386 22715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fe88462cf9754df18cbc3e92e8ffda56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "29c43aba55024b2997ea2cb1686a2250"
I20260926 01:14:18.903472 22715 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 29c43aba55024b2997ea2cb1686a2250 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:14:18.904367 22715 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 29c43aba55024b2997ea2cb1686a2250 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe88462cf9754df18cbc3e92e8ffda56 in term 1.
I20260926 01:14:18.904606 22714 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 29c43aba55024b2997ea2cb1686a2250 [term 1 FOLLOWER]: Refusing update from remote peer fe88462cf9754df18cbc3e92e8ffda56: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:14:18.905602 22822 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fe88462cf9754df18cbc3e92e8ffda56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29c43aba55024b2997ea2cb1686a2250" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:14:18.903321 22866 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29c43aba55024b2997ea2cb1686a2250 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe88462cf9754df18cbc3e92e8ffda56" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36503 } } peers { permanent_uuid: "29c43aba55024b2997ea2cb1686a2250" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34597 } } peers { permanent_uuid: "fad13ee3df594713bf1a627c9a4a8175" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41699 } } }
I20260926 01:14:18.908631 22866 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29c43aba55024b2997ea2cb1686a2250 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 01:14:18.908738 22866 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29c43aba55024b2997ea2cb1686a2250 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fe88462cf9754df18cbc3e92e8ffda56. Latest consensus state: current_term: 1 leader_uuid: "fe88462cf9754df18cbc3e92e8ffda56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe88462cf9754df18cbc3e92e8ffda56" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36503 } } peers { permanent_uuid: "29c43aba55024b2997ea2cb1686a2250" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34597 } } peers { permanent_uuid: "fad13ee3df594713bf1a627c9a4a8175" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41699 } } }
I20260926 01:14:18.908840 22866 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29c43aba55024b2997ea2cb1686a2250 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 01:14:18.908901 22866 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29c43aba55024b2997ea2cb1686a2250 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fe88462cf9754df18cbc3e92e8ffda56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe88462cf9754df18cbc3e92e8ffda56" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36503 } } peers { permanent_uuid: "29c43aba55024b2997ea2cb1686a2250" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34597 } } peers { permanent_uuid: "fad13ee3df594713bf1a627c9a4a8175" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41699 } } }
I20260926 01:14:18.908988 22866 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29c43aba55024b2997ea2cb1686a2250 [sys.catalog]: This master's current role is: FOLLOWER
W20260926 01:14:18.909243 22877 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 29c43aba55024b2997ea2cb1686a2250: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 01:14:18.909291 22877 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 01:14:18.911551 22879 mvcc.cc:205] Tried to move back new op lower bound from 7333418020388237312 to 7333418020211138560. Current Snapshot: MvccSnapshot[applied={T|T < 7333418020220977152}]
W20260926 01:14:18.924746 22669 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:14:18.923574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39730 (local address 127.0.98.62:36503)
0926 01:14:18.923617 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 01:14:18.923619 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:14:18.924057 (+ 438us) server_negotiation.cc:407] Connection header received
0926 01:14:18.924127 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:14:18.924128 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:14:18.924140 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:14:18.924215 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:14:18.924681 (+ 466us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39730: BlockingRecv error: recv got EOF from 127.0.0.1:39730 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260926 01:14:18.924787 22819 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:14:18.923773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49308 (local address 127.0.98.60:41699)
0926 01:14:18.923814 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 01:14:18.923816 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:14:18.924195 (+ 379us) server_negotiation.cc:407] Connection header received
0926 01:14:18.924283 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:14:18.924284 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:14:18.924294 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:14:18.924322 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:14:18.924721 (+ 399us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49308: BlockingRecv error: recv got EOF from 127.0.0.1:49308 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260926 01:14:18.925499 22746 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:14:18.923308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49400 (local address 127.0.98.61:34597)
0926 01:14:18.923353 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 01:14:18.923356 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:14:18.924972 (+ 1616us) server_negotiation.cc:407] Connection header received
0926 01:14:18.925065 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:14:18.925067 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:14:18.925079 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:14:18.925108 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:14:18.925437 (+ 329us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49400: BlockingRecv error: recv got EOF from 127.0.0.1:49400 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260926 01:14:18.931885 22746 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:14:18.931034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49410 (local address 127.0.98.61:34597)
0926 01:14:18.931078 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 01:14:18.931080 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:14:18.931278 (+ 198us) server_negotiation.cc:407] Connection header received
0926 01:14:18.931385 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:14:18.931387 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:14:18.931410 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:14:18.931438 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:14:18.931827 (+ 389us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49410: BlockingRecv error: recv got EOF from 127.0.0.1:49410 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260926 01:14:18.935148 22746 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:14:18.934264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49414 (local address 127.0.98.61:34597)
0926 01:14:18.934303 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 01:14:18.934305 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:14:18.934556 (+ 251us) server_negotiation.cc:407] Connection header received
0926 01:14:18.934660 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:14:18.934661 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:14:18.934672 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:14:18.934701 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:14:18.935087 (+ 386us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49414: BlockingRecv error: recv got EOF from 127.0.0.1:49414 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260926 01:14:18.939414 22746 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:14:18.938505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49424 (local address 127.0.98.61:34597)
0926 01:14:18.938557 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 01:14:18.938559 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:14:18.938831 (+ 272us) server_negotiation.cc:407] Connection header received
0926 01:14:18.938935 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:14:18.938936 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:14:18.938947 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:14:18.938976 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:14:18.939354 (+ 378us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49424: BlockingRecv error: recv got EOF from 127.0.0.1:49424 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
W20260926 01:14:18.945734 22746 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:14:18.944803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49438 (local address 127.0.98.61:34597)
0926 01:14:18.944847 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 01:14:18.944849 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:14:18.945115 (+ 266us) server_negotiation.cc:407] Connection header received
0926 01:14:18.945219 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:14:18.945221 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:14:18.945233 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:14:18.945263 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:14:18.945668 (+ 405us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49438: BlockingRecv error: recv got EOF from 127.0.0.1:49438 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260926 01:14:18.956195 22746 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:14:18.955251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49454 (local address 127.0.98.61:34597)
0926 01:14:18.955298 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 01:14:18.955300 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:14:18.955572 (+ 272us) server_negotiation.cc:407] Connection header received
0926 01:14:18.955679 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:14:18.955680 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:14:18.955693 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:14:18.955723 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:14:18.956121 (+ 398us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49454: BlockingRecv error: recv got EOF from 127.0.0.1:49454 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260926 01:14:18.974576 22746 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:14:18.973611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49460 (local address 127.0.98.61:34597)
0926 01:14:18.973655 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 01:14:18.973657 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:14:18.973958 (+ 301us) server_negotiation.cc:407] Connection header received
0926 01:14:18.974069 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:14:18.974070 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:14:18.974082 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:14:18.974112 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:14:18.974502 (+ 390us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49460: BlockingRecv error: recv got EOF from 127.0.0.1:49460 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260926 01:14:19.008456 22746 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:14:19.007795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49468 (local address 127.0.98.61:34597)
0926 01:14:19.007844 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 01:14:19.007847 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:14:19.007954 (+ 107us) server_negotiation.cc:407] Connection header received
0926 01:14:19.008009 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:14:19.008011 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:14:19.008032 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:14:19.008080 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:14:19.008393 (+ 313us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49468: BlockingRecv error: recv got EOF from 127.0.0.1:49468 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260926 01:14:19.074277 22746 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:14:19.073631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49480 (local address 127.0.98.61:34597)
0926 01:14:19.073733 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0926 01:14:19.073735 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:14:19.073748 (+ 13us) server_negotiation.cc:407] Connection header received
0926 01:14:19.073789 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:14:19.073790 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:14:19.073801 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:14:19.073866 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:14:19.074213 (+ 347us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49480: BlockingRecv error: recv got EOF from 127.0.0.1:49480 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260926 01:14:19.204155 22746 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:14:19.203442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49488 (local address 127.0.98.61:34597)
0926 01:14:19.203519 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0926 01:14:19.203522 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:14:19.203569 (+ 47us) server_negotiation.cc:407] Connection header received
0926 01:14:19.203648 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:14:19.203649 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:14:19.203660 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:14:19.203709 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:14:19.204075 (+ 366us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49488: BlockingRecv error: recv got EOF from 127.0.0.1:49488 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 01:14:19.461012 22939 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe88462cf9754df18cbc3e92e8ffda56 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fe88462cf9754df18cbc3e92e8ffda56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe88462cf9754df18cbc3e92e8ffda56" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36503 } } peers { permanent_uuid: "29c43aba55024b2997ea2cb1686a2250" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34597 } } peers { permanent_uuid: "fad13ee3df594713bf1a627c9a4a8175" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41699 } } }
I20260926 01:14:19.461161 22939 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe88462cf9754df18cbc3e92e8ffda56 [sys.catalog]: This master's current role is: LEADER
W20260926 01:14:19.461892 22746 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:14:19.461290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49494 (local address 127.0.98.61:34597)
0926 01:14:19.461334 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 01:14:19.461336 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:14:19.461409 (+ 73us) server_negotiation.cc:407] Connection header received
0926 01:14:19.461465 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:14:19.461465 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:14:19.461475 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:14:19.461520 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:14:19.461840 (+ 320us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49494: BlockingRecv error: recv got EOF from 127.0.0.1:49494 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 01:14:19.850297 22857 catalog_manager.cc:1259] Loaded cluster ID: dba77eb4674b4ae5af696916f2e6fb93
I20260926 01:14:19.850378 22857 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fad13ee3df594713bf1a627c9a4a8175: loading cluster ID for follower catalog manager: success
I20260926 01:14:19.851204 22857 catalog_manager.cc:1574] T 00000000000000000000000000000000 P fad13ee3df594713bf1a627c9a4a8175: acquiring CA information for follower catalog manager: success
I20260926 01:14:19.851480 22857 catalog_manager.cc:1602] T 00000000000000000000000000000000 P fad13ee3df594713bf1a627c9a4a8175: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 01:14:19.909747 22877 catalog_manager.cc:1259] Loaded cluster ID: dba77eb4674b4ae5af696916f2e6fb93
I20260926 01:14:19.909855 22877 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 29c43aba55024b2997ea2cb1686a2250: loading cluster ID for follower catalog manager: success
I20260926 01:14:19.910647 22877 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 29c43aba55024b2997ea2cb1686a2250: acquiring CA information for follower catalog manager: success
I20260926 01:14:19.910902 22877 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 29c43aba55024b2997ea2cb1686a2250: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 01:14:19.977751 22954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:14:19.975195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49506 (local address 127.0.98.61:34597)
0926 01:14:19.975296 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0926 01:14:19.975298 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:14:19.975340 (+ 42us) server_negotiation.cc:407] Connection header received
0926 01:14:19.975426 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:14:19.975427 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:14:19.975479 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:14:19.975545 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:14:19.975690 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:14:19.976187 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:14:19.977003 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:14:19.977202 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:14:19.977257 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:14:19.977419 (+ 162us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:14:19.977532 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:14:19.977600 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 01:14:19.977638 (+ 38us) server_negotiation.cc:299] Negotiation successful
0926 01:14:19.977661 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79,"thread_start_us":49,"threads_started":1}
I20260926 01:14:19.982241 22960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:14:19.979604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49310 (local address 127.0.98.60:41699)
0926 01:14:19.979729 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0926 01:14:19.979731 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:14:19.979741 (+ 10us) server_negotiation.cc:407] Connection header received
0926 01:14:19.979780 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:14:19.979780 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:14:19.979816 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:14:19.979891 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:14:19.979983 (+ 92us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:14:19.980460 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:14:19.981253 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:14:19.981475 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:14:19.981543 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:14:19.981805 (+ 262us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:14:19.981982 (+ 177us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:14:19.982053 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 01:14:19.982131 (+ 78us) server_negotiation.cc:299] Negotiation successful
0926 01:14:19.982153 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":58,"threads_started":1}
I20260926 01:14:19.987241 22954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:14:19.984139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49508 (local address 127.0.98.61:34597)
0926 01:14:19.984185 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 01:14:19.984187 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:14:19.984200 (+ 13us) server_negotiation.cc:407] Connection header received
0926 01:14:19.984314 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:14:19.984315 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:14:19.984328 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:14:19.984406 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:14:19.984640 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:14:19.985420 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:14:19.986515 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:14:19.986758 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:14:19.986817 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:14:19.986987 (+ 170us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:14:19.987017 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:14:19.987052 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0926 01:14:19.987122 (+ 70us) server_negotiation.cc:299] Negotiation successful
0926 01:14:19.987145 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 01:14:19.987468 22967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:14:19.984162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39746 (local address 127.0.98.62:36503)
0926 01:14:19.984266 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0926 01:14:19.984268 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:14:19.984351 (+ 83us) server_negotiation.cc:407] Connection header received
0926 01:14:19.984439 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:14:19.984441 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:14:19.984499 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:14:19.984577 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:14:19.984709 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:14:19.985255 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:14:19.986402 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:14:19.986711 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:14:19.986766 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:14:19.986998 (+ 232us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:14:19.987120 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:14:19.987183 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 01:14:19.987356 (+ 173us) server_negotiation.cc:299] Negotiation successful
0926 01:14:19.987385 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86,"thread_start_us":51,"threads_started":1}
I20260926 01:14:19.987646 22960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:14:19.984368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49318 (local address 127.0.98.60:41699)
0926 01:14:19.984466 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0926 01:14:19.984468 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:14:19.984475 (+ 7us) server_negotiation.cc:407] Connection header received
0926 01:14:19.984544 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:14:19.984545 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:14:19.984556 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:14:19.984603 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:14:19.984764 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:14:19.985499 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:14:19.986302 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:14:19.986584 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:14:19.986660 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:14:19.986831 (+ 171us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:14:19.986869 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:14:19.986923 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 01:14:19.987285 (+ 362us) server_negotiation.cc:299] Negotiation successful
0926 01:14:19.987313 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 01:14:19.989063 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 22598
I20260926 01:14:19.993934 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 22673
I20260926 01:14:19.998685 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 22748
2026-09-26T01:14:20Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2736 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384993574762-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 01:14:20 dist-test-slave-7b22 krb5kdc[22971](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 01:14:20 dist-test-slave-7b22 krb5kdc[22971](info): set up 2 sockets
Sep 26 01:14:20 dist-test-slave-7b22 krb5kdc[22971](info): commencing operation
krb5kdc: starting...
W20260926 01:14:20.966285 392 mini_kdc.cc:121] Time spent starting KDC: real 0.948s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384993574762-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384993574762-392-0/krb5kdc/test-user.keytab.
Sep 26 01:14:21 dist-test-slave-7b22 krb5kdc[22971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385261, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T01:14:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:14:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384993574762-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384993574762-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 01:14:21.093412 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384993574762-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384993574762-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384993574762-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42353
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384993574762-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42353,127.0.98.61:36663,127.0.98.60:37645
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:14:21.179114 22984 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 01:14:21.179358 22984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:14:21.179382 22984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:14:21.179410 22984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:14:21.180642 22984 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:14:21.180696 22984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:14:21.180714 22984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:14:21.180730 22984 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:14:21.180744 22984 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:14:21.182394 22984 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384993574762-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384993574762-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42353,127.0.98.61:36663,127.0.98.60:37645
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384993574762-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42353
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22984
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384993574762-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:14:21.182680 22984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:14:21.182914 22984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:14:21.187824 22990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:14:21.187829 22989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:14:21.187990 22984 server_base.cc:1034] running on GCE node
W20260926 01:14:21.188195 22992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:14:21.188414 22984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:14:21.188730 22984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:14:21.189899 22984 hybrid_clock.cc:648] HybridClock initialized: now 1790385261189872 us; error 40 us; skew 500 ppm
Sep 26 01:14:21 dist-test-slave-7b22 krb5kdc[22971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385261, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:14:21.192378 22984 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 01:14:21.192837 22984 webserver.cc:460] Webserver started at http://127.0.98.62:39179/ using document root <none> and password file <none>
I20260926 01:14:21.193058 22984 fs_manager.cc:359] Metadata directory not provided
I20260926 01:14:21.193109 22984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:14:21.193239 22984 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:14:21.194418 22984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384993574762-392-0/minicluster-data/master-0/data/instance:
uuid: "8e37cbdccc724e3e9dfb644e6cc8889f"
format_stamp: "Formatted at 2026-09-26 01:14:21 on dist-test-slave-7b22"
server_key: "8e7d276aec32dd13d1d2f87f040e8d04"
server_key_iv: "3e133bd25c31d1d9835cea6ecdeff46e"
server_key_version: "encryptionkey@0"
I20260926 01:14:21.194761 22984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384993574762-392-0/minicluster-data/master-0/wal/instance:
uuid: "8e37cbdccc724e3e9dfb644e6cc8889f"
format_stamp: "Formatted at 2026-09-26 01:14:21 on dist-test-slave-7b22"
server_key: "8e7d276aec32dd13d1d2f87f040e8d04"
server_key_iv: "3e133bd25c31d1d9835cea6ecdeff46e"
server_key_version: "encryptionkey@0"
I20260926 01:14:21.196135 22984 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 01:14:21.196817 22999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:14:21.196949 22984 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 01:14:21.197036 22984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384993574762-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384993574762-392-0/minicluster-data/master-0/wal
uuid: "8e37cbdccc724e3e9dfb644e6cc8889f"
format_stamp: "Formatted at 2026-09-26 01:14:21 on dist-test-slave-7b22"
server_key: "8e7d276aec32dd13d1d2f87f040e8d04"
server_key_iv: "3e133bd25c31d1d9835cea6ecdeff46e"
server_key_version: "encryptionkey@0"
I20260926 01:14:21.197139 22984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384993574762-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384993574762-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384993574762-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:14:21.213940 22984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:14:21.216565 22984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:14:21.216717 22984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:14:21.220494 22984 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42353
I20260926 01:14:21.220607 23051 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42353 every 8 connection(s)
I20260926 01:14:21.220868 22984 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384993574762-392-0/minicluster-data/master-0/data/info.pb
I20260926 01:14:21.221968 23052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:14:21.223616 23052 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42353 } has no permanent_uuid. Determining permanent_uuid...
I20260926 01:14:21.229305 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 22984
I20260926 01:14:21.229426 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384993574762-392-0/minicluster-data/master-0/wal/instance
I20260926 01:14:21.229611 392 external_mini_cluster.cc:1428] Setting key a4570d40c618f739fbf8d2552e24a72e
Sep 26 01:14:21 dist-test-slave-7b22 krb5kdc[22971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385261, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:14:21.234355 23054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:14:21.224348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42353 (local address 127.0.0.1:44518)
0926 01:14:21.224543 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0926 01:14:21.224580 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 01:14:21.225063 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:14:21.225364 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:14:21.225367 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:14:21.225677 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:14:21.225879 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:14:21.225885 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:14:21.226915 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:14:21.226916 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:14:21.227629 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:14:21.227631 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:14:21.228137 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:14:21.228138 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:14:21.228197 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:14:21.228688 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:14:21.228697 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:14:21.230467 (+ 1770us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:14:21.232598 (+ 2131us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:14:21.232605 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:14:21.232607 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:14:21.232810 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:14:21.233100 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:14:21.233101 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:14:21.233101 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:14:21.233216 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:14:21.233675 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:14:21.233681 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:14:21.233751 (+ 70us) client_negotiation.cc:769] Sending connection context
0926 01:14:21.233957 (+ 206us) client_negotiation.cc:240] Negotiation successful
0926 01:14:21.234116 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":84,"threads_started":1}
I20260926 01:14:21.234362 23055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:14:21.224382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44518 (local address 127.0.98.62:42353)
0926 01:14:21.224528 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0926 01:14:21.224538 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 01:14:21.224653 (+ 115us) server_negotiation.cc:407] Connection header received
0926 01:14:21.225203 (+ 550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:14:21.225205 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:14:21.225223 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:14:21.225311 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:14:21.225997 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:14:21.226807 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:14:21.227715 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:14:21.228039 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:14:21.228120 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:14:21.230589 (+ 2469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:14:21.230596 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:14:21.230596 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:14:21.230603 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:14:21.232502 (+ 1899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:14:21.232888 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:14:21.232891 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:14:21.232891 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:14:21.233004 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:14:21.233283 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:14:21.233284 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:14:21.233285 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:14:21.233594 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:14:21.233683 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 01:14:21.234004 (+ 321us) server_negotiation.cc:299] Negotiation successful
0926 01:14:21.234183 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":64,"server-negotiator.queue_time_us":105,"thread_start_us":66,"threads_started":1}
I20260926 01:14:21.235219 23052 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36663 } has no permanent_uuid. Determining permanent_uuid...
I20260926 01:14:21.235591 23054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:14:21.235465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36663 (local address 127.0.0.1:53446)
0926 01:14:21.235512 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0926 01:14:21.235559 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260926 01:14:21.235697 23003 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36663: connect: Connection refused (error 111)
I20260926 01:14:21.236130 23054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:14:21.236031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36663 (local address 127.0.0.1:53460)
0926 01:14:21.236074 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0926 01:14:21.236104 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260926 01:14:21.236294 23052 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:36663: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36663: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384993574762-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384993574762-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260926 01:14:21.258116 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384993574762-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384993574762-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384993574762-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384993574762-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:36663
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384993574762-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42353,127.0.98.61:36663,127.0.98.60:37645
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 01:14:21.285540 23052 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36663 } attempt: 1
I20260926 01:14:21.286088 23054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:14:21.285919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36663 (local address 127.0.0.1:53472)
0926 01:14:21.285988 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0926 01:14:21.286048 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260926 01:14:21.286494 23054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:14:21.286387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36663 (local address 127.0.0.1:53486)
0926 01:14:21.286435 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0926 01:14:21.286469 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260926 01:14:21.286597 23052 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:36663: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36663: connect: Connection refused (error 111)
W20260926 01:14:21.344367 23059 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 01:14:21.344630 23059 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:14:21.344650 23059 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:14:21.344678 23059 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:14:21.346525 23059 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:14:21.346582 23059 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:14:21.346596 23059 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:14:21.346607 23059 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:14:21.346618 23059 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:14:21.348345 23059 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384993574762-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384993574762-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42353,127.0.98.61:36663,127.0.98.60:37645
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384993574762-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:36663
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384993574762-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23059
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384993574762-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:14:21.348615 23059 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:14:21.348855 23059 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:14:21.353681 23064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:14:21.353704 23065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:14:21.353899 23059 server_base.cc:1034] running on GCE node
W20260926 01:14:21.353693 23067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:14:21.354224 23059 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:14:21.354503 23059 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:14:21.354739 23052 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36663 } attempt: 2
I20260926 01:14:21.355083 23054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:14:21.354965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36663 (local address 127.0.0.1:53494)
0926 01:14:21.355015 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0926 01:14:21.355046 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260926 01:14:21.355377 23054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:14:21.355298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36663 (local address 127.0.0.1:53504)
0926 01:14:21.355333 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0926 01:14:21.355355 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260926 01:14:21.355540 23052 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:36663: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36663: connect: Connection refused (error 111)
I20260926 01:14:21.355777 23059 hybrid_clock.cc:648] HybridClock initialized: now 1790385261355754 us; error 28 us; skew 500 ppm
Sep 26 01:14:21 dist-test-slave-7b22 krb5kdc[22971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385261, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:14:21.358527 23059 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260926 01:14:21.359018 23059 webserver.cc:460] Webserver started at http://127.0.98.61:42141/ using document root <none> and password file <none>
I20260926 01:14:21.359236 23059 fs_manager.cc:359] Metadata directory not provided
I20260926 01:14:21.359279 23059 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:14:21.359401 23059 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:14:21.360462 23059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384993574762-392-0/minicluster-data/master-1/data/instance:
uuid: "b36ef30122ff4658a35c5a88d6dda18a"
format_stamp: "Formatted at 2026-09-26 01:14:21 on dist-test-slave-7b22"
server_key: "fefc7e156e9a68341d85a6c38d2b69d4"
server_key_iv: "4db4f80a09aa3efc14926f874b31fd77"
server_key_version: "encryptionkey@0"
I20260926 01:14:21.360785 23059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384993574762-392-0/minicluster-data/master-1/wal/instance:
uuid: "b36ef30122ff4658a35c5a88d6dda18a"
format_stamp: "Formatted at 2026-09-26 01:14:21 on dist-test-slave-7b22"
server_key: "fefc7e156e9a68341d85a6c38d2b69d4"
server_key_iv: "4db4f80a09aa3efc14926f874b31fd77"
server_key_version: "encryptionkey@0"
I20260926 01:14:21.362222 23059 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 01:14:21.363216 23074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:14:21.363425 23059 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:14:21.363509 23059 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384993574762-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384993574762-392-0/minicluster-data/master-1/wal
uuid: "b36ef30122ff4658a35c5a88d6dda18a"
format_stamp: "Formatted at 2026-09-26 01:14:21 on dist-test-slave-7b22"
server_key: "fefc7e156e9a68341d85a6c38d2b69d4"
server_key_iv: "4db4f80a09aa3efc14926f874b31fd77"
server_key_version: "encryptionkey@0"
I20260926 01:14:21.363600 23059 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384993574762-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384993574762-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384993574762-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:14:21.375721 23059 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:14:21.377787 23059 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:14:21.377942 23059 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:14:21.381570 23059 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:36663
I20260926 01:14:21.381629 23126 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:36663 every 8 connection(s)
I20260926 01:14:21.381970 23059 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384993574762-392-0/minicluster-data/master-1/data/info.pb
I20260926 01:14:21.382936 23127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:14:21.383315 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 23059
I20260926 01:14:21.383406 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384993574762-392-0/minicluster-data/master-1/wal/instance
I20260926 01:14:21.383574 392 external_mini_cluster.cc:1428] Setting key d4d6543f44b0421e37af8ce9a70143fe
I20260926 01:14:21.384701 23127 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42353 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 01:14:21 dist-test-slave-7b22 krb5kdc[22971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385261, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
I20260926 01:14:21.394833 23055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:14:21.385400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44528 (local address 127.0.98.62:42353)
0926 01:14:21.385465 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 01:14:21.385467 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:14:21.385691 (+ 224us) server_negotiation.cc:407] Connection header received
0926 01:14:21.386187 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:14:21.386188 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:14:21.386203 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:14:21.386281 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:14:21.387001 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:14:21.387532 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:14:21.388510 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:14:21.388712 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:14:21.388769 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:14:21.391606 (+ 2837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:14:21.391609 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:14:21.391609 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:14:21.391617 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:14:21.392978 (+ 1361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:14:21.393516 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:14:21.393517 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:14:21.393518 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:14:21.393579 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:14:21.393870 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:14:21.393870 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:14:21.393871 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:14:21.394053 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:14:21.394130 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 01:14:21.394632 (+ 502us) server_negotiation.cc:299] Negotiation successful
0926 01:14:21.394727 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 01:14:21.395210 23130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:14:21.385368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42353 (local address 127.0.0.1:44528)
0926 01:14:21.385534 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 01:14:21.385582 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 01:14:21.386080 (+ 498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:14:21.386401 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:14:21.386405 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:14:21.386678 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:14:21.386895 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:14:21.386901 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:14:21.387616 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:14:21.387617 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:14:21.388425 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:14:21.388428 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:14:21.388804 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:14:21.388805 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:14:21.388934 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:14:21.389570 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:14:21.389579 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:14:21.391415 (+ 1836us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:14:21.393107 (+ 1692us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:14:21.393119 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:14:21.393122 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:14:21.393447 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:14:21.393646 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:14:21.393646 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:14:21.393647 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:14:21.393794 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:14:21.394198 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:14:21.394212 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:14:21.394440 (+ 228us) client_negotiation.cc:769] Sending connection context
0926 01:14:21.394574 (+ 134us) client_negotiation.cc:240] Negotiation successful
0926 01:14:21.394932 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":78,"threads_started":1}
I20260926 01:14:21.395939 23127 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36663 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 01:14:21 dist-test-slave-7b22 krb5kdc[22971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385261, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260926 01:14:21.404179 23130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:14:21.396201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36663 (local address 127.0.0.1:53518)
0926 01:14:21.396249 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0926 01:14:21.396260 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 01:14:21.396312 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:14:21.396597 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:14:21.396598 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:14:21.396731 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:14:21.396792 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:14:21.396794 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:14:21.398131 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:14:21.398132 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:14:21.398682 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:14:21.398684 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:14:21.399384 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:14:21.399385 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:14:21.399472 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:14:21.399991 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:14:21.399996 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:14:21.401460 (+ 1464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:14:21.403212 (+ 1752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:14:21.403213 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:14:21.403214 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:14:21.403367 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:14:21.403538 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:14:21.403539 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:14:21.403539 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:14:21.403587 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:14:21.403837 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:14:21.403838 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:14:21.403877 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 01:14:21.403936 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 01:14:21.404048 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260926 01:14:21.404189 23132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:14:21.396260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53518 (local address 127.0.98.61:36663)
0926 01:14:21.396417 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0926 01:14:21.396422 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 01:14:21.396443 (+ 21us) server_negotiation.cc:407] Connection header received
0926 01:14:21.396466 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:14:21.396475 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:14:21.396491 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:14:21.396567 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:14:21.396863 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:14:21.398031 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:14:21.398778 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:14:21.399294 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:14:21.399378 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:14:21.401549 (+ 2171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:14:21.401562 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:14:21.401562 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:14:21.401570 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:14:21.403123 (+ 1553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:14:21.403435 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:14:21.403439 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:14:21.403439 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:14:21.403487 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:14:21.403634 (+ 147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:14:21.403634 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:14:21.403635 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:14:21.403782 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:14:21.403851 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 01:14:21.403969 (+ 118us) server_negotiation.cc:299] Negotiation successful
0926 01:14:21.404094 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":63,"server-negotiator.queue_time_us":127,"thread_start_us":93,"threads_started":1}
I20260926 01:14:21.404995 23127 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37645 } has no permanent_uuid. Determining permanent_uuid...
I20260926 01:14:21.405357 23130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:14:21.405238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37645 (local address 127.0.0.1:37914)
0926 01:14:21.405280 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0926 01:14:21.405333 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260926 01:14:21.405489 23077 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37645: connect: Connection refused (error 111)
I20260926 01:14:21.405900 23130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:14:21.405785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37645 (local address 127.0.0.1:37930)
0926 01:14:21.405851 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0926 01:14:21.405876 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260926 01:14:21.405990 23127 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37645: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37645: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384993574762-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384993574762-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260926 01:14:21.408844 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzzHEKz/build/release/bin/kudu
/tmp/dist-test-taskzzHEKz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384993574762-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384993574762-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384993574762-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384993574762-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:37645
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384993574762-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42353,127.0.98.61:36663,127.0.98.60:37645
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384993574762-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384993574762-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384993574762-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 01:14:21.446789 23052 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36663 } attempt: 3
Sep 26 01:14:21 dist-test-slave-7b22 krb5kdc[22971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385261, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260926 01:14:21.455053 23132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:14:21.447436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53522 (local address 127.0.98.61:36663)
0926 01:14:21.447608 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 01:14:21.447611 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:14:21.447629 (+ 18us) server_negotiation.cc:407] Connection header received
0926 01:14:21.447665 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:14:21.447666 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:14:21.447686 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:14:21.447870 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:14:21.448279 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:14:21.448795 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:14:21.449799 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:14:21.450074 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:14:21.450132 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:14:21.452280 (+ 2148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:14:21.452281 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:14:21.452281 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:14:21.452289 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:14:21.453655 (+ 1366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:14:21.454141 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:14:21.454144 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:14:21.454145 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:14:21.454269 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:14:21.454499 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:14:21.454499 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:14:21.454500 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:14:21.454636 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:14:21.454698 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 01:14:21.454861 (+ 163us) server_negotiation.cc:299] Negotiation successful
0926 01:14:21.454966 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 01:14:21.455081 23054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:14:21.447349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36663 (local address 127.0.0.1:53522)
0926 01:14:21.447438 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0926 01:14:21.447453 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 01:14:21.447517 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:14:21.447892 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:14:21.447893 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:14:21.448090 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:14:21.448166 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:14:21.448167 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:14:21.448874 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:14:21.448875 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:14:21.449703 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:14:21.449707 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:14:21.450139 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:14:21.450140 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:14:21.450218 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:14:21.450716 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:14:21.450721 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:14:21.452165 (+ 1444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:14:21.453745 (+ 1580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:14:21.453746 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:14:21.453747 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:14:21.454002 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:14:21.454352 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:14:21.454352 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:14:21.454353 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:14:21.454425 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:14:21.454696 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:14:21.454697 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:14:21.454756 (+ 59us) client_negotiation.cc:769] Sending connection context
0926 01:14:21.454841 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 01:14:21.454957 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260926 01:14:21.455603 23127 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37645 } attempt: 1
I20260926 01:14:21.455749 23052 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37645 } has no permanent_uuid. Determining permanent_uuid...
I20260926 01:14:21.456043 23130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:14:21.455909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37645 (local address 127.0.0.1:37938)
0926 01:14:21.455960 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0926 01:14:21.456004 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260926 01:14:21.456061 23054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:14:21.455938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37645 (local address 127.0.0.1:37950)
0926 01:14:21.456010 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0926 01:14:21.456039 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260926 01:14:21.456372 23054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:14:21.456268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37645 (local address 127.0.0.1:37962)
0926 01:14:21.456312 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0926 01:14:21.456349 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260926 01:14:21.456374 23130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:14:21.456299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37645 (local address 127.0.0.1:37978)
0926 01:14:21.456330 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0926 01:14:21.456354 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
W20260926 01:14:21.456456 23052 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37645: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37645: connect: Connection refused (error 111)
W20260926 01:14:21.456475 23127 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37645: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37645: connect: Connection refused (error 111)
I20260926 01:14:21.487622 23052 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37645 } attempt: 1
I20260926 01:14:21.488160 23054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:14:21.488031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37645 (local address 127.0.0.1:37994)
0926 01:14:21.488094 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0926 01:14:21.488127 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260926 01:14:21.488571 23054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:14:21.488446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37645 (local address 127.0.0.1:38006)
0926 01:14:21.488514 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0926 01:14:21.488548 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260926 01:14:21.488668 23052 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37645: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37645: connect: Connection refused (error 111)
W20260926 01:14:21.493813 23134 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 01:14:21.494097 23134 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:14:21.494115 23134 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:14:21.494138 23134 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:14:21.495399 23134 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:14:21.495447 23134 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:14:21.495460 23134 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:14:21.495481 23134 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:14:21.495492 23134 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:14:21.497032 23134 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384993574762-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384993574762-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42353,127.0.98.61:36663,127.0.98.60:37645
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384993574762-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:37645
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384993574762-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23134
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384993574762-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:14:21.497277 23134 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:14:21.497505 23134 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:14:21.502588 23140 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:14:21.502584 23142 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:14:21.502655 23134 server_base.cc:1034] running on GCE node
W20260926 01:14:21.502581 23139 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:14:21.503486 23134 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:14:21.503823 23134 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:14:21.504988 23134 hybrid_clock.cc:648] HybridClock initialized: now 1790385261504972 us; error 34 us; skew 500 ppm
Sep 26 01:14:21 dist-test-slave-7b22 krb5kdc[22971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385261, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:14:21.507761 23134 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260926 01:14:21.508332 23134 webserver.cc:460] Webserver started at http://127.0.98.60:44537/ using document root <none> and password file <none>
I20260926 01:14:21.508543 23134 fs_manager.cc:359] Metadata directory not provided
I20260926 01:14:21.508590 23134 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:14:21.508721 23134 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:14:21.509814 23134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384993574762-392-0/minicluster-data/master-2/data/instance:
uuid: "a71e80c4014043f2ac1117ff361028a5"
format_stamp: "Formatted at 2026-09-26 01:14:21 on dist-test-slave-7b22"
server_key: "071af3b8344f9b6aa4b0a9814264bc4d"
server_key_iv: "4601fff370d8a2448abb7252c5c6b7db"
server_key_version: "encryptionkey@0"
I20260926 01:14:21.510217 23134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384993574762-392-0/minicluster-data/master-2/wal/instance:
uuid: "a71e80c4014043f2ac1117ff361028a5"
format_stamp: "Formatted at 2026-09-26 01:14:21 on dist-test-slave-7b22"
server_key: "071af3b8344f9b6aa4b0a9814264bc4d"
server_key_iv: "4601fff370d8a2448abb7252c5c6b7db"
server_key_version: "encryptionkey@0"
I20260926 01:14:21.511965 23134 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260926 01:14:21.512831 23149 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:14:21.513036 23134 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:14:21.513126 23134 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384993574762-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384993574762-392-0/minicluster-data/master-2/wal
uuid: "a71e80c4014043f2ac1117ff361028a5"
format_stamp: "Formatted at 2026-09-26 01:14:21 on dist-test-slave-7b22"
server_key: "071af3b8344f9b6aa4b0a9814264bc4d"
server_key_iv: "4601fff370d8a2448abb7252c5c6b7db"
server_key_version: "encryptionkey@0"
I20260926 01:14:21.513199 23134 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384993574762-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384993574762-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384993574762-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:14:21.524659 23127 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37645 } attempt: 2
I20260926 01:14:21.525177 23130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:14:21.525010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37645 (local address 127.0.0.1:38010)
0926 01:14:21.525081 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0926 01:14:21.525132 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260926 01:14:21.525520 23130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:14:21.525439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37645 (local address 127.0.0.1:38022)
0926 01:14:21.525479 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0926 01:14:21.525503 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260926 01:14:21.525650 23127 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37645: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37645: connect: Connection refused (error 111)
I20260926 01:14:21.529717 23134 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:14:21.531803 23134 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:14:21.531949 23134 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:14:21.535990 23134 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:37645
I20260926 01:14:21.536026 23201 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:37645 every 8 connection(s)
I20260926 01:14:21.536379 23134 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384993574762-392-0/minicluster-data/master-2/data/info.pb
I20260926 01:14:21.537495 23202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:14:21.539209 23202 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42353 } has no permanent_uuid. Determining permanent_uuid...
I20260926 01:14:21.544097 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzzHEKz/build/release/bin/kudu as pid 23134
I20260926 01:14:21.544185 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzzHEKz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384993574762-392-0/minicluster-data/master-2/wal/instance
I20260926 01:14:21.544370 392 external_mini_cluster.cc:1428] Setting key 2d30d9921e65b1408e9a83ab684e9667
Sep 26 01:14:21 dist-test-slave-7b22 krb5kdc[22971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385261, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:14:21.549101 23204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:14:21.539954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42353 (local address 127.0.0.1:44544)
0926 01:14:21.540154 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0926 01:14:21.540195 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 01:14:21.540714 (+ 519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:14:21.540976 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:14:21.540979 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:14:21.541255 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:14:21.541475 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:14:21.541480 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:14:21.542217 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:14:21.542218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:14:21.543086 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:14:21.543089 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:14:21.543514 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:14:21.543515 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:14:21.543634 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:14:21.544214 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:14:21.544225 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:14:21.545852 (+ 1627us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:14:21.547511 (+ 1659us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:14:21.547518 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:14:21.547520 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:14:21.547853 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:14:21.548062 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:14:21.548062 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:14:21.548063 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:14:21.548195 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:14:21.548474 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:14:21.548479 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:14:21.548647 (+ 168us) client_negotiation.cc:769] Sending connection context
0926 01:14:21.548748 (+ 101us) client_negotiation.cc:240] Negotiation successful
0926 01:14:21.548879 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":103,"threads_started":1}
I20260926 01:14:21.549100 23055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:14:21.540014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44544 (local address 127.0.98.62:42353)
0926 01:14:21.540118 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0926 01:14:21.540121 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:14:21.540302 (+ 181us) server_negotiation.cc:407] Connection header received
0926 01:14:21.540824 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:14:21.540825 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:14:21.540836 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:14:21.540913 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:14:21.541570 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:14:21.542118 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:14:21.543209 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:14:21.543400 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:14:21.543464 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:14:21.545923 (+ 2459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:14:21.545925 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:14:21.545925 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:14:21.545933 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:14:21.547414 (+ 1481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:14:21.547935 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:14:21.547936 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:14:21.547936 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:14:21.548009 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:14:21.548263 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:14:21.548264 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:14:21.548264 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:14:21.548406 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:14:21.548470 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 01:14:21.548824 (+ 354us) server_negotiation.cc:299] Negotiation successful
0926 01:14:21.548999 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 01:14:21.549723 23202 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36663 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 01:14:21 dist-test-slave-7b22 krb5kdc[22971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385261, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 26 01:14:21 dist-test-slave-7b22 krb5kdc[22971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385261, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260926 01:14:21.555382 23206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:14:21.544783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38026 (local address 127.0.98.60:37645)
0926 01:14:21.544952 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 01:14:21.544957 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 01:14:21.544981 (+ 24us) server_negotiation.cc:407] Connection header received
0926 01:14:21.545005 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:14:21.545008 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:14:21.545030 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:14:21.545112 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:14:21.545781 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:14:21.546656 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:14:21.547663 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:14:21.548105 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:14:21.548184 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:14:21.550909 (+ 2725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:14:21.550919 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:14:21.550919 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:14:21.550927 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:14:21.553331 (+ 2404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:14:21.554223 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:14:21.554232 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:14:21.554232 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:14:21.554318 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:14:21.554556 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:14:21.554557 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:14:21.554557 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:14:21.554823 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:14:21.555015 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0926 01:14:21.555172 (+ 157us) server_negotiation.cc:299] Negotiation successful
0926 01:14:21.555264 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":91,"threads_started":1}
I20260926 01:14:21.557238 23204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:14:21.550031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36663 (local address 127.0.0.1:53528)
0926 01:14:21.550116 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0926 01:14:21.550128 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 01:14:21.550175 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:14:21.550292 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:14:21.550293 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:14:21.550429 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:14:21.550491 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:14:21.550493 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:14:21.551262 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:14:21.551262 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:14:21.551787 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:14:21.551789 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:14:21.552207 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:14:21.552208 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:14:21.552300 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:14:21.552711 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:14:21.552717 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:14:21.554355 (+ 1638us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:14:21.556146 (+ 1791us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:14:21.556147 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:14:21.556147 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:14:21.556353 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:14:21.556549 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:14:21.556550 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:14:21.556550 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:14:21.556612 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:14:21.556836 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:14:21.556837 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:14:21.556959 (+ 122us) client_negotiation.cc:769] Sending connection context
0926 01:14:21.557019 (+ 60us) client_negotiation.cc:240] Negotiation successful
0926 01:14:21.557135 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260926 01:14:21.557238 23132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:14:21.550142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53528 (local address 127.0.98.61:36663)
0926 01:14:21.550197 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 01:14:21.550200 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:14:21.550208 (+ 8us) server_negotiation.cc:407] Connection header received
0926 01:14:21.550233 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:14:21.550234 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:14:21.550244 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:14:21.550289 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:14:21.550591 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:14:21.551200 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:14:21.551864 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:14:21.552131 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:14:21.552197 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:14:21.554470 (+ 2273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:14:21.554472 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:14:21.554473 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:14:21.554485 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:14:21.556087 (+ 1602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:14:21.556422 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:14:21.556423 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:14:21.556423 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:14:21.556487 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:14:21.556663 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:14:21.556664 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:14:21.556665 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:14:21.556783 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:14:21.556838 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 01:14:21.557059 (+ 221us) server_negotiation.cc:299] Negotiation successful
0926 01:14:21.557135 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 01:14:21.558137 23202 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37645 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 01:14:21 dist-test-slave-7b22 krb5kdc[22971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385261, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 26 01:14:21 dist-test-slave-7b22 krb5kdc[22971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385261, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260926 01:14:21.563848 23052 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37645 } attempt: 2
I20260926 01:14:21.564796 23204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:14:21.558433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37645 (local address 127.0.0.1:38028)
0926 01:14:21.558483 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0926 01:14:21.558495 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 01:14:21.558542 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:14:21.558654 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:14:21.558655 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:14:21.558787 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:14:21.558822 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:14:21.558823 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:14:21.559643 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:14:21.559644 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:14:21.560124 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:14:21.560126 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:14:21.560464 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:14:21.560464 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:14:21.560510 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:14:21.560920 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:14:21.560925 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:14:21.562032 (+ 1107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:14:21.563691 (+ 1659us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:14:21.563692 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:14:21.563693 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:14:21.563892 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:14:21.564138 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:14:21.564139 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:14:21.564139 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:14:21.564199 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:14:21.564475 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:14:21.564475 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:14:21.564512 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 01:14:21.564568 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 01:14:21.564670 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260926 01:14:21.564999 23055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:14:21.556522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44560 (local address 127.0.98.62:42353)
0926 01:14:21.556565 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 01:14:21.556567 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:14:21.556574 (+ 7us) server_negotiation.cc:407] Connection header received
0926 01:14:21.556596 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:14:21.556596 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:14:21.556607 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:14:21.556655 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:14:21.557107 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:14:21.557718 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:14:21.558612 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:14:21.558819 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:14:21.558875 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:14:21.561238 (+ 2363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:14:21.561240 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:14:21.561240 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:14:21.561248 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:14:21.562767 (+ 1519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:14:21.564156 (+ 1389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:14:21.564158 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:14:21.564158 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:14:21.564216 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:14:21.564447 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:14:21.564448 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:14:21.564448 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:14:21.564582 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:14:21.564792 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0926 01:14:21.564835 (+ 43us) server_negotiation.cc:299] Negotiation successful
0926 01:14:21.564923 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 01:14:21.564867 23206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:14:21.558493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38028 (local address 127.0.98.60:37645)
0926 01:14:21.558525 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0926 01:14:21.558527 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:14:21.558534 (+ 7us) server_negotiation.cc:407] Connection header received
0926 01:14:21.558594 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:14:21.558594 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:14:21.558604 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:14:21.558644 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:14:21.558932 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:14:21.559566 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:14:21.560191 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:14:21.560409 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:14:21.560462 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:14:21.562109 (+ 1647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:14:21.562110 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:14:21.562110 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:14:21.562117 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:14:21.563591 (+ 1474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:14:21.563988 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:14:21.563989 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:14:21.563989 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:14:21.564056 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:14:21.564257 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:14:21.564258 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:14:21.564258 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:14:21.564398 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:14:21.564455 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 01:14:21.564616 (+ 161us) server_negotiation.cc:299] Negotiation successful
0926 01:14:21.564730 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":19}
I20260926 01:14:21.566931 23202 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a71e80c4014043f2ac1117ff361028a5: Bootstrap starting.
I20260926 01:14:21.567936 23202 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a71e80c4014043f2ac1117ff361028a5: Neither blocks nor log segments found. Creating new log.
I20260926 01:14:21.568256 23202 log.cc:824] T 00000000000000000000000000000000 P a71e80c4014043f2ac1117ff361028a5: Log is configured to *not* fsync() on all Append() calls
I20260926 01:14:21.569243 23202 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a71e80c4014043f2ac1117ff361028a5: No bootstrap required, opened a new log
Sep 26 01:14:21 dist-test-slave-7b22 krb5kdc[22971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385261, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 26 01:14:21 dist-test-slave-7b22 krb5kdc[22971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385261, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260926 01:14:21.571216 23202 raft_consensus.cc:348] T 00000000000000000000000000000000 P a71e80c4014043f2ac1117ff361028a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e37cbdccc724e3e9dfb644e6cc8889f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42353 } } peers { permanent_uuid: "b36ef30122ff4658a35c5a88d6dda18a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36663 } } peers { permanent_uuid: "a71e80c4014043f2ac1117ff361028a5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37645 } }
I20260926 01:14:21.571507 23202 raft_consensus.cc:729] T 00000000000000000000000000000000 P a71e80c4014043f2ac1117ff361028a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a71e80c4014043f2ac1117ff361028a5, State: Initialized, Role: FOLLOWER
I20260926 01:14:21.571624 23202 consensus_queue.cc:260] T 00000000000000000000000000000000 P a71e80c4014043f2ac1117ff361028a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e37cbdccc724e3e9dfb644e6cc8889f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42353 } } peers { permanent_uuid: "b36ef30122ff4658a35c5a88d6dda18a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36663 } } peers { permanent_uuid: "a71e80c4014043f2ac1117ff361028a5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37645 } }
I20260926 01:14:21.572033 23209 sys_catalog.cc:455] T 00000000000000000000000000000000 P a71e80c4014043f2ac1117ff361028a5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e37cbdccc724e3e9dfb644e6cc8889f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42353 } } peers { permanent_uuid: "b36ef30122ff4658a35c5a88d6dda18a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36663 } } peers { permanent_uuid: "a71e80c4014043f2ac1117ff361028a5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37645 } } }
I20260926 01:14:21.572094 23202 sys_catalog.cc:564] T 00000000000000000000000000000000 P a71e80c4014043f2ac1117ff361028a5 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:14:21.573089 23206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:14:21.564290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38042 (local address 127.0.98.60:37645)
0926 01:14:21.565184 (+ 894us) server_negotiation.cc:206] Beginning negotiation
0926 01:14:21.565186 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:14:21.565195 (+ 9us) server_negotiation.cc:407] Connection header received
0926 01:14:21.565216 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:14:21.565217 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:14:21.565229 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:14:21.565305 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:14:21.565946 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:14:21.566609 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:14:21.567429 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:14:21.567700 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:14:21.567769 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:14:21.570142 (+ 2373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:14:21.570143 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:14:21.570144 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:14:21.570151 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:14:21.571718 (+ 1567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:14:21.572206 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:14:21.572208 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:14:21.572208 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:14:21.572267 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:14:21.572468 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:14:21.572469 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:14:21.572469 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:14:21.572637 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:14:21.572706 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 01:14:21.572914 (+ 208us) server_negotiation.cc:299] Negotiation successful
0926 01:14:21.573000 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":878,"thread_start_us":125,"threads_started":1}
I20260926 01:14:21.573107 23054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:14:21.564160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37645 (local address 127.0.0.1:38042)
0926 01:14:21.564205 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0926 01:14:21.564217 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 01:14:21.564252 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:14:21.565318 (+ 1066us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:14:21.565319 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:14:21.565478 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:14:21.565863 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:14:21.565866 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:14:21.566678 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:14:21.566679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:14:21.567339 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:14:21.567342 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:14:21.567776 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:14:21.567777 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:14:21.567863 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:14:21.568356 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:14:21.568361 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:14:21.570028 (+ 1667us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:14:21.571912 (+ 1884us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:14:21.571914 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:14:21.571914 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:14:21.572132 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:14:21.572329 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:14:21.572329 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:14:21.572330 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:14:21.572400 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:14:21.572741 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:14:21.572742 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:14:21.572799 (+ 57us) client_negotiation.cc:769] Sending connection context
0926 01:14:21.572882 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 01:14:21.573005 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260926 01:14:21.573628 23209 sys_catalog.cc:458] T 00000000000000000000000000000000 P a71e80c4014043f2ac1117ff361028a5 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 01:14:21.574213 23132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:14:21.565631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53536 (local address 127.0.98.61:36663)
0926 01:14:21.565674 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 01:14:21.565675 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 01:14:21.565682 (+ 7us) server_negotiation.cc:407] Connection header received
0926 01:14:21.565705 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:14:21.565706 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:14:21.565716 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:14:21.565768 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:14:21.566416 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:14:21.567107 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:14:21.567881 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:14:21.568162 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:14:21.568232 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:14:21.570712 (+ 2480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:14:21.570713 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:14:21.570713 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:14:21.570721 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:14:21.572484 (+ 1763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:14:21.572847 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:14:21.572848 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:14:21.572849 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:14:21.572917 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:14:21.573114 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:14:21.573115 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:14:21.573115 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:14:21.573254 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:14:21.573390 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0926 01:14:21.574034 (+ 644us) server_negotiation.cc:299] Negotiation successful
0926 01:14:21.574132 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
W20260926 01:14:21.574333 23220 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a71e80c4014043f2ac1117ff361028a5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 01:14:21.574369 23220 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 01:14:21.575052 23052 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e37cbdccc724e3e9dfb644e6cc8889f: Bootstrap starting.
I20260926 01:14:21.575676 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 01:14:21.575855 23052 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8e37cbdccc724e3e9dfb644e6cc8889f: Neither blocks nor log segments found. Creating new log.
I20260926 01:14:21.576156 23052 log.cc:824] T 00000000000000000000000000000000 P 8e37cbdccc724e3e9dfb644e6cc8889f: Log is configured to *not* fsync() on all Append() calls
I20260926 01:14:21.574311 23134 master_runner.cc:186] Error getting master registration for 127.0.98.62:42353: OK, instance_id { permanent_uuid: "8e37cbdccc724e3e9dfb644e6cc8889f" instance_seqno: 1790385261219452 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 01:14:21.576738 23134 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 01:14:21.577412 23052 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e37cbdccc724e3e9dfb644e6cc8889f: No bootstrap required, opened a new log
I20260926 01:14:21.576653 23026 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "8e37cbdccc724e3e9dfb644e6cc8889f"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44560
I20260926 01:14:21.578781 23052 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8e37cbdccc724e3e9dfb644e6cc8889f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e37cbdccc724e3e9dfb644e6cc8889f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42353 } } peers { permanent_uuid: "b36ef30122ff4658a35c5a88d6dda18a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36663 } } peers { permanent_uuid: "a71e80c4014043f2ac1117ff361028a5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37645 } }
I20260926 01:14:21.579211 23052 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8e37cbdccc724e3e9dfb644e6cc8889f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e37cbdccc724e3e9dfb644e6cc8889f, State: Initialized, Role: FOLLOWER
I20260926 01:14:21.579516 23052 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8e37cbdccc724e3e9dfb644e6cc8889f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e37cbdccc724e3e9dfb644e6cc8889f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42353 } } peers { permanent_uuid: "b36ef30122ff4658a35c5a88d6dda18a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36663 } } peers { permanent_uuid: "a71e80c4014043f2ac1117ff361028a5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37645 } }
I20260926 01:14:21.580018 23052 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8e37cbdccc724e3e9dfb644e6cc8889f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:14:21.580046 23026 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8e37cbdccc724e3e9dfb644e6cc8889f [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260926 01:14:21.580103 23026 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8e37cbdccc724e3e9dfb644e6cc8889f [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:14:21.581189 23026 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8e37cbdccc724e3e9dfb644e6cc8889f [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e37cbdccc724e3e9dfb644e6cc8889f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42353 } } peers { permanent_uuid: "b36ef30122ff4658a35c5a88d6dda18a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36663 } } peers { permanent_uuid: "a71e80c4014043f2ac1117ff361028a5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37645 } }
I20260926 01:14:21.582238 23026 leader_election.cc:288] T 00000000000000000000000000000000 P 8e37cbdccc724e3e9dfb644e6cc8889f [CANDIDATE]: Term 1 election: Requested vote from peers b36ef30122ff4658a35c5a88d6dda18a (127.0.98.61:36663), a71e80c4014043f2ac1117ff361028a5 (127.0.98.60:37645)
I20260926 01:14:21.582599 23176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8e37cbdccc724e3e9dfb644e6cc8889f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "a71e80c4014043f2ac1117ff361028a5"
I20260926 01:14:21.582746 23176 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a71e80c4014043f2ac1117ff361028a5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:14:21.584220 23176 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a71e80c4014043f2ac1117ff361028a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e37cbdccc724e3e9dfb644e6cc8889f in term 1.
I20260926 01:14:21.584527 23002 leader_election.cc:302] T 00000000000000000000000000000000 P 8e37cbdccc724e3e9dfb644e6cc8889f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e37cbdccc724e3e9dfb644e6cc8889f, a71e80c4014043f2ac1117ff361028a5; no voters:
I20260926 01:14:21.584157 22984 master_runner.cc:186] Error getting master registration for 127.0.98.61:36663: OK, instance_id { permanent_uuid: "b36ef30122ff4658a35c5a88d6dda18a" instance_seqno: 1790385261380545 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 01:14:21.588063 22984 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260926 01:14:21.582269 23234 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8e37cbdccc724e3e9dfb644e6cc8889f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 01:14:21.588176 23234 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 01:14:21.588307 23243 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e37cbdccc724e3e9dfb644e6cc8889f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e37cbdccc724e3e9dfb644e6cc8889f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42353 } } peers { permanent_uuid: "b36ef30122ff4658a35c5a88d6dda18a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36663 } } peers { permanent_uuid: "a71e80c4014043f2ac1117ff361028a5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37645 } } }
I20260926 01:14:21.588411 23243 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e37cbdccc724e3e9dfb644e6cc8889f [sys.catalog]: This master's current role is: FOLLOWER
I20260926 01:14:21.588479 23243 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8e37cbdccc724e3e9dfb644e6cc8889f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:14:21.588541 23243 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8e37cbdccc724e3e9dfb644e6cc8889f [term 1 LEADER]: Becoming Leader. State: Replica: 8e37cbdccc724e3e9dfb644e6cc8889f, State: Running, Role: LEADER
I20260926 01:14:21.588647 23243 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8e37cbdccc724e3e9dfb644e6cc8889f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e37cbdccc724e3e9dfb644e6cc8889f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42353 } } peers { permanent_uuid: "b36ef30122ff4658a35c5a88d6dda18a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36663 } } peers { permanent_uuid: "a71e80c4014043f2ac1117ff361028a5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37645 } }
I20260926 01:14:21.589171 23243 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e37cbdccc724e3e9dfb644e6cc8889f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8e37cbdccc724e3e9dfb644e6cc8889f. Latest consensus state: current_term: 1 leader_uuid: "8e37cbdccc724e3e9dfb644e6cc8889f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e37cbdccc724e3e9dfb644e6cc8889f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42353 } } peers { permanent_uuid: "b36ef30122ff4658a35c5a88d6dda18a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36663 } } peers { permanent_uuid: "a71e80c4014043f2ac1117ff361028a5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37645 } } }
I20260926 01:14:21.589268 23243 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e37cbdccc724e3e9dfb644e6cc8889f [sys.catalog]: This master's current role is: LEADER
I20260926 01:14:21.593246 23206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:14:21.583650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38050 (local address 127.0.98.60:37645)
0926 01:14:21.583702 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 01:14:21.583705 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:14:21.584798 (+ 1093us) server_negotiation.cc:407] Connection header received
0926 01:14:21.584907 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:14:21.584908 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:14:21.584920 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:14:21.584952 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:14:21.585842 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:14:21.586383 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:14:21.587482 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:14:21.587700 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:14:21.587762 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:14:21.589404 (+ 1642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:14:21.589406 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:14:21.589406 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:14:21.589415 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:14:21.591531 (+ 2116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:14:21.592106 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:14:21.592108 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:14:21.592108 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:14:21.592184 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:14:21.592487 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:14:21.592488 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:14:21.592488 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:14:21.592636 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:14:21.592708 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 01:14:21.593047 (+ 339us) server_negotiation.cc:299] Negotiation successful
0926 01:14:21.593145 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 01:14:21.593943 23245 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:14:21.594234 23245 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:14:21.598344 23055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:14:21.584113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44576 (local address 127.0.98.62:42353)
0926 01:14:21.584169 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 01:14:21.584171 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:14:21.589793 (+ 5622us) server_negotiation.cc:407] Connection header received
0926 01:14:21.590305 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:14:21.590309 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:14:21.590324 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:14:21.590392 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:14:21.591116 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:14:21.591841 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:14:21.592816 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:14:21.593111 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:14:21.593152 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:14:21.595582 (+ 2430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:14:21.595583 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:14:21.595583 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:14:21.595592 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:14:21.596993 (+ 1401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:14:21.597402 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:14:21.597403 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:14:21.597403 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:14:21.597466 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:14:21.597660 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:14:21.597661 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:14:21.597661 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:14:21.597806 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:14:21.597923 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0926 01:14:21.598147 (+ 224us) server_negotiation.cc:299] Negotiation successful
0926 01:14:21.598244 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 01:14:21.598495 23132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:14:21.584462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53548 (local address 127.0.98.61:36663)
0926 01:14:21.584527 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 01:14:21.584529 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:14:21.590026 (+ 5497us) server_negotiation.cc:407] Connection header received
0926 01:14:21.590367 (+ 341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:14:21.590369 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:14:21.590387 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:14:21.590421 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:14:21.591228 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:14:21.591997 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:14:21.593077 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:14:21.593395 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:14:21.593438 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:14:21.595249 (+ 1811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:14:21.595251 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:14:21.595251 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:14:21.595259 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:14:21.596822 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:14:21.597480 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:14:21.597481 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:14:21.597481 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:14:21.597548 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:14:21.597812 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:14:21.597813 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:14:21.597813 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:14:21.597972 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:14:21.598035 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 01:14:21.598321 (+ 286us) server_negotiation.cc:299] Negotiation successful
0926 01:14:21.598397 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 01:14:21.607904 23176 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a71e80c4014043f2ac1117ff361028a5 [term 1 FOLLOWER]: Refusing update from remote peer 8e37cbdccc724e3e9dfb644e6cc8889f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:14:21.608408 23223 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8e37cbdccc724e3e9dfb644e6cc8889f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a71e80c4014043f2ac1117ff361028a5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:14:21.610333 23209 sys_catalog.cc:455] T 00000000000000000000000000000000 P a71e80c4014043f2ac1117ff361028a5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8e37cbdccc724e3e9dfb644e6cc8889f. Latest consensus state: current_term: 1 leader_uuid: "8e37cbdccc724e3e9dfb644e6cc8889f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e37cbdccc724e3e9dfb644e6cc8889f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42353 } } peers { permanent_uuid: "b36ef30122ff4658a35c5a88d6dda18a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36663 } } peers { permanent_uuid: "a71e80c4014043f2ac1117ff361028a5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37645 } } }
I20260926 01:14:21.610460 23209 sys_catalog.cc:458] T 00000000000000000000000000000000 P a71e80c4014043f2ac1117ff361028a5 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 01:14:21.616767 23127 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37645 } attempt: 3
I20260926 01:14:21.618426 23223 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e37cbdccc724e3e9dfb644e6cc8889f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8e37cbdccc724e3e9dfb644e6cc8889f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e37cbdccc724e3e9dfb644e6cc8889f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42353 } } peers { permanent_uuid: "b36ef30122ff4658a35c5a88d6dda18a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36663 } } peers { permanent_uuid: "a71e80c4014043f2ac1117ff361028a5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37645 } } }
I20260926 01:14:21.618585 23223 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e37cbdccc724e3e9dfb644e6cc8889f [sys.catalog]: This master's current role is: LEADER
I20260926 01:14:21.618911 23246 mvcc.cc:205] Tried to move back new op lower bound from 7333418031541633024 to 7333418031467954176. Current Snapshot: MvccSnapshot[applied={T|T < 7333418031541633024}]
I20260926 01:14:21.619194 23209 sys_catalog.cc:455] T 00000000000000000000000000000000 P a71e80c4014043f2ac1117ff361028a5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8e37cbdccc724e3e9dfb644e6cc8889f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e37cbdccc724e3e9dfb644e6cc8889f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42353 } } peers { permanent_uuid: "b36ef30122ff4658a35c5a88d6dda18a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36663 } } peers { permanent_uuid: "a71e80c4014043f2ac1117ff361028a5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37645 } } }
I20260926 01:14:21.619300 23209 sys_catalog.cc:458] T 00000000000000000000000000000000 P a71e80c4014043f2ac1117ff361028a5 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 01:14:21.619571 23245 catalog_manager.cc:1347] Generated new cluster ID: ec7c3cf118064890b6c01d355f31177f
I20260926 01:14:21.619622 23245 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 01:14:21 dist-test-slave-7b22 krb5kdc[22971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790385261, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260926 01:14:21.625895 23130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:14:21.617159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37645 (local address 127.0.0.1:38062)
0926 01:14:21.617224 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0926 01:14:21.617238 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 01:14:21.617287 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:14:21.617433 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:14:21.617434 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:14:21.617653 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:14:21.617702 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:14:21.617704 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:14:21.618689 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:14:21.618690 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:14:21.619444 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:14:21.619448 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:14:21.620079 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:14:21.620080 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:14:21.620141 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:14:21.620668 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:14:21.620673 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:14:21.622832 (+ 2159us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:14:21.624702 (+ 1870us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:14:21.624703 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:14:21.624704 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:14:21.624926 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:14:21.625155 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:14:21.625156 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:14:21.625157 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:14:21.625218 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:14:21.625498 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:14:21.625499 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:14:21.625572 (+ 73us) client_negotiation.cc:769] Sending connection context
0926 01:14:21.625636 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 01:14:21.625755 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260926 01:14:21.626312 23206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:14:21.617201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38062 (local address 127.0.98.60:37645)
0926 01:14:21.617275 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0926 01:14:21.617277 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:14:21.617287 (+ 10us) server_negotiation.cc:407] Connection header received
0926 01:14:21.617365 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:14:21.617366 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:14:21.617378 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:14:21.617428 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:14:21.617813 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:14:21.618515 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:14:21.619592 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:14:21.619864 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:14:21.619948 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:14:21.622914 (+ 2966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:14:21.622915 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:14:21.622916 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:14:21.622924 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:14:21.624594 (+ 1670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:14:21.625007 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:14:21.625008 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:14:21.625008 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:14:21.625093 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:14:21.625274 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:14:21.625275 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:14:21.625276 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:14:21.625433 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:14:21.625505 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 01:14:21.626128 (+ 623us) server_negotiation.cc:299] Negotiation successful
0926 01:14:21.626216 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 01:14:21.628149 23127 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b36ef30122ff4658a35c5a88d6dda18a: Bootstrap starting.
I20260926 01:14:21.629078 23127 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b36ef30122ff4658a35c5a88d6dda18a: Neither blocks nor log segments found. Creating new log.
I20260926 01:14:21.629408 23127 log.cc:824] T 00000000000000000000000000000000 P b36ef30122ff4658a35c5a88d6dda18a: Log is configured to *not* fsync() on all Append() calls
I20260926 01:14:21.635325 23127 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b36ef30122ff4658a35c5a88d6dda18a: No bootstrap required, opened a new log
I20260926 01:14:21.635704 23247 mvcc.cc:205] Tried to move back new op lower bound from 7333418031541633024 to 7333418031467954176. Current Snapshot: MvccSnapshot[applied={T|T < 7333418031541633024 or (T in {7333418031541633024})}]
I20260926 01:14:21.637691 23127 raft_consensus.cc:348] T 00000000000000000000000000000000 P b36ef30122ff4658a35c5a88d6dda18a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e37cbdccc724e3e9dfb644e6cc8889f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42353 } } peers { permanent_uuid: "b36ef30122ff4658a35c5a88d6dda18a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36663 } } peers { permanent_uuid: "a71e80c4014043f2ac1117ff361028a5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37645 } }
I20260926 01:14:21.637912 23127 raft_consensus.cc:729] T 00000000000000000000000000000000 P b36ef30122ff4658a35c5a88d6dda18a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b36ef30122ff4658a35c5a88d6dda18a, State: Initialized, Role: FOLLOWER
I20260926 01:14:21.638007 23127 consensus_queue.cc:260] T 00000000000000000000000000000000 P b36ef30122ff4658a35c5a88d6dda18a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e37cbdccc724e3e9dfb644e6cc8889f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42353 } } peers { permanent_uuid: "b36ef30122ff4658a35c5a88d6dda18a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36663 } } peers { permanent_uuid: "a71e80c4014043f2ac1117ff361028a5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37645 } }
I20260926 01:14:21.638388 23127 sys_catalog.cc:564] T 00000000000000000000000000000000 P b36ef30122ff4658a35c5a88d6dda18a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:14:21.640026 23101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8e37cbdccc724e3e9dfb644e6cc8889f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b36ef30122ff4658a35c5a88d6dda18a"
I20260926 01:14:21.640213 23100 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b36ef30122ff4658a35c5a88d6dda18a [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:14:21.643226 23100 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b36ef30122ff4658a35c5a88d6dda18a [term 1 FOLLOWER]: Refusing update from remote peer 8e37cbdccc724e3e9dfb644e6cc8889f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:14:21.643910 23223 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8e37cbdccc724e3e9dfb644e6cc8889f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b36ef30122ff4658a35c5a88d6dda18a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:14:21.644615 23101 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b36ef30122ff4658a35c5a88d6dda18a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e37cbdccc724e3e9dfb644e6cc8889f in term 1.
I20260926 01:14:21.645870 23245 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:14:21.646405 23264 sys_catalog.cc:455] T 00000000000000000000000000000000 P b36ef30122ff4658a35c5a88d6dda18a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8e37cbdccc724e3e9dfb644e6cc8889f. Latest consensus state: current_term: 1 leader_uuid: "8e37cbdccc724e3e9dfb644e6cc8889f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e37cbdccc724e3e9dfb644e6cc8889f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42353 } } peers { permanent_uuid: "b36ef30122ff4658a35c5a88d6dda18a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36663 } } peers { permanent_uuid: "a71e80c4014043f2ac1117ff361028a5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37645 } } }
I20260926 01:14:21.646548 23264 sys_catalog.cc:458] T 00000000000000000000000000000000 P b36ef30122ff4658a35c5a88d6dda18a [sys.catalog]: This master's current role is: FOLLOWER
I20260926 01:14:21.646836 23245 catalog_manager.cc:1504] Loading token signing keys...
W20260926 01:14:21.647236 23263 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b36ef30122ff4658a35c5a88d6dda18a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 01:14:21.647338 23263 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 01:14:21.647509 23252 sys_catalog.cc:455] T 00000000000000000000000000000000 P b36ef30122ff4658a35c5a88d6dda18a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8e37cbdccc724e3e9dfb644e6cc8889f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e37cbdccc724e3e9dfb644e6cc8889f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42353 } } peers { permanent_uuid: "b36ef30122ff4658a35c5a88d6dda18a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36663 } } peers { permanent_uuid: "a71e80c4014043f2ac1117ff361028a5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37645 } } }
I20260926 01:14:21.647635 23252 sys_catalog.cc:458] T 00000000000000000000000000000000 P b36ef30122ff4658a35c5a88d6dda18a [sys.catalog]: This master's current role is: FOLLOWER
I20260926 01:14:21.646250 23267 sys_catalog.cc:455] T 00000000000000000000000000000000 P b36ef30122ff4658a35c5a88d6dda18a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8e37cbdccc724e3e9dfb644e6cc8889f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e37cbdccc724e3e9dfb644e6cc8889f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42353 } } peers { permanent_uuid: "b36ef30122ff4658a35c5a88d6dda18a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36663 } } peers { permanent_uuid: "a71e80c4014043f2ac1117ff361028a5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37645 } } }
I20260926 01:14:21.648103 23267 sys_catalog.cc:458] T 00000000000000000000000000000000 P b36ef30122ff4658a35c5a88d6dda18a [sys.catalog]: This master's current role is: FOLLOWER
I20260926 01:14:21.651036 23223 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e37cbdccc724e3e9dfb644e6cc8889f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8e37cbdccc724e3e9dfb644e6cc8889f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e37cbdccc724e3e9dfb644e6cc8889f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42353 } } peers { permanent_uuid: "b36ef30122ff4658a35c5a88d6dda18a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36663 } } peers { permanent_uuid: "a71e80c4014043f2ac1117ff361028a5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37645 } } }
I20260926 01:14:21.651141 23223 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e37cbdccc724e3e9dfb644e6cc8889f [sys.catalog]: This master's current role is: LEADER
I20260926 01:14:21.652411 23245 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8e37cbdccc724e3e9dfb644e6cc8889f: Generated new TSK 0
I20260926 01:14:21.652611 23245 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 01:14:21.674126 23055 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:14:21.672292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44584 (local address 127.0.98.62:42353)
0926 01:14:21.672356 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 01:14:21.672358 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:14:21.673341 (+ 983us) server_negotiation.cc:407] Connection header received
0926 01:14:21.673453 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:14:21.673454 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:14:21.673466 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:14:21.673496 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:14:21.674033 (+ 537us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44584: BlockingRecv error: recv got EOF from 127.0.0.1:44584 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260926 01:14:21.675182 23132 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:14:21.672680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53552 (local address 127.0.98.61:36663)
0926 01:14:21.672722 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 01:14:21.672724 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:14:21.674348 (+ 1624us) server_negotiation.cc:407] Connection header received
0926 01:14:21.674446 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:14:21.674447 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:14:21.674460 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:14:21.674487 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:14:21.675101 (+ 614us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53552: BlockingRecv error: recv got EOF from 127.0.0.1:53552 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260926 01:14:21.675923 23206 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:14:21.673022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38074 (local address 127.0.98.60:37645)
0926 01:14:21.673072 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 01:14:21.673074 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:14:21.675384 (+ 2310us) server_negotiation.cc:407] Connection header received
0926 01:14:21.675505 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:14:21.675506 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:14:21.675518 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:14:21.675548 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:14:21.675855 (+ 307us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38074: BlockingRecv error: recv got EOF from 127.0.0.1:38074 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
W20260926 01:14:21.681517 23132 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:14:21.680620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53556 (local address 127.0.98.61:36663)
0926 01:14:21.680667 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 01:14:21.680670 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:14:21.680801 (+ 131us) server_negotiation.cc:407] Connection header received
0926 01:14:21.680974 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:14:21.680975 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:14:21.681000 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:14:21.681030 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:14:21.681444 (+ 414us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53556: BlockingRecv error: recv got EOF from 127.0.0.1:53556 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260926 01:14:21.689172 23132 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:14:21.688217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53570 (local address 127.0.98.61:36663)
0926 01:14:21.688266 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 01:14:21.688269 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:14:21.688570 (+ 301us) server_negotiation.cc:407] Connection header received
0926 01:14:21.688672 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:14:21.688674 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:14:21.688686 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:14:21.688715 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:14:21.689106 (+ 391us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53570: BlockingRecv error: recv got EOF from 127.0.0.1:53570 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
W20260926 01:14:21.693284 23132 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:14:21.692385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53572 (local address 127.0.98.61:36663)
0926 01:14:21.692429 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 01:14:21.692431 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:14:21.692704 (+ 273us) server_negotiation.cc:407] Connection header received
0926 01:14:21.692804 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:14:21.692805 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:14:21.692818 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:14:21.692847 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:14:21.693218 (+ 371us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53572: BlockingRecv error: recv got EOF from 127.0.0.1:53572 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260926 01:14:21.699589 23132 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:14:21.698598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53584 (local address 127.0.98.61:36663)
0926 01:14:21.698643 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 01:14:21.698645 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:14:21.698938 (+ 293us) server_negotiation.cc:407] Connection header received
0926 01:14:21.699036 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:14:21.699037 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:14:21.699049 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:14:21.699076 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:14:21.699527 (+ 451us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53584: BlockingRecv error: recv got EOF from 127.0.0.1:53584 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260926 01:14:21.710070 23132 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:14:21.708990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53592 (local address 127.0.98.61:36663)
0926 01:14:21.709041 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 01:14:21.709044 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:14:21.709346 (+ 302us) server_negotiation.cc:407] Connection header received
0926 01:14:21.709471 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:14:21.709473 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:14:21.709485 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:14:21.709515 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:14:21.709986 (+ 471us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53592: BlockingRecv error: recv got EOF from 127.0.0.1:53592 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260926 01:14:21.728631 23132 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:14:21.727530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53602 (local address 127.0.98.61:36663)
0926 01:14:21.727594 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 01:14:21.727596 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:14:21.727944 (+ 348us) server_negotiation.cc:407] Connection header received
0926 01:14:21.728061 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:14:21.728062 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:14:21.728077 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:14:21.728108 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:14:21.728552 (+ 444us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53602: BlockingRecv error: recv got EOF from 127.0.0.1:53602 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260926 01:14:21.763154 23132 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:14:21.762086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53614 (local address 127.0.98.61:36663)
0926 01:14:21.762137 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 01:14:21.762139 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:14:21.762503 (+ 364us) server_negotiation.cc:407] Connection header received
0926 01:14:21.762618 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:14:21.762619 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:14:21.762632 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:14:21.762662 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:14:21.763087 (+ 425us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53614: BlockingRecv error: recv got EOF from 127.0.0.1:53614 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260926 01:14:21.829948 23132 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:14:21.829064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53618 (local address 127.0.98.61:36663)
0926 01:14:21.829141 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0926 01:14:21.829143 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:14:21.829155 (+ 12us) server_negotiation.cc:407] Connection header received
0926 01:14:21.829252 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:14:21.829254 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:14:21.829267 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:14:21.829313 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:14:21.829874 (+ 561us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53618: BlockingRecv error: recv got EOF from 127.0.0.1:53618 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260926 01:14:21.960086 23132 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:14:21.959490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53624 (local address 127.0.98.61:36663)
0926 01:14:21.959586 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0926 01:14:21.959588 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:14:21.959596 (+ 8us) server_negotiation.cc:407] Connection header received
0926 01:14:21.959621 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:14:21.959622 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:14:21.959633 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:14:21.959681 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:14:21.960035 (+ 354us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53624: BlockingRecv error: recv got EOF from 127.0.0.1:53624 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260926 01:14:22.218472 23132 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:14:22.217842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53638 (local address 127.0.98.61:36663)
0926 01:14:22.217904 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 01:14:22.217906 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:14:22.217915 (+ 9us) server_negotiation.cc:407] Connection header received
0926 01:14:22.217942 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:14:22.217943 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:14:22.217953 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:14:22.218010 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:14:22.218419 (+ 409us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53638: BlockingRecv error: recv got EOF from 127.0.0.1:53638 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
I20260926 01:14:22.574831 23220 catalog_manager.cc:1259] Loaded cluster ID: ec7c3cf118064890b6c01d355f31177f
I20260926 01:14:22.575112 23220 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a71e80c4014043f2ac1117ff361028a5: loading cluster ID for follower catalog manager: success
I20260926 01:14:22.576205 23220 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a71e80c4014043f2ac1117ff361028a5: acquiring CA information for follower catalog manager: success
I20260926 01:14:22.576529 23220 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a71e80c4014043f2ac1117ff361028a5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 01:14:22.647944 23263 catalog_manager.cc:1259] Loaded cluster ID: ec7c3cf118064890b6c01d355f31177f
I20260926 01:14:22.648031 23263 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b36ef30122ff4658a35c5a88d6dda18a: loading cluster ID for follower catalog manager: success
I20260926 01:14:22.648844 23263 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b36ef30122ff4658a35c5a88d6dda18a: acquiring CA information for follower catalog manager: success
I20260926 01:14:22.649156 23263 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b36ef30122ff4658a35c5a88d6dda18a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 01:14:22.734388 23337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:14:22.731787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53650 (local address 127.0.98.61:36663)
0926 01:14:22.731891 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0926 01:14:22.731894 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:14:22.732002 (+ 108us) server_negotiation.cc:407] Connection header received
0926 01:14:22.732074 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:14:22.732075 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:14:22.732131 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:14:22.732199 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:14:22.732335 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:14:22.732841 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:14:22.733605 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:14:22.733803 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:14:22.733890 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:14:22.734046 (+ 156us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:14:22.734161 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:14:22.734227 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 01:14:22.734275 (+ 48us) server_negotiation.cc:299] Negotiation successful
0926 01:14:22.734297 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80,"thread_start_us":45,"threads_started":1}
I20260926 01:14:22.739732 23343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:14:22.736613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38078 (local address 127.0.98.60:37645)
0926 01:14:22.736742 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0926 01:14:22.736744 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:14:22.736754 (+ 10us) server_negotiation.cc:407] Connection header received
0926 01:14:22.736793 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:14:22.736793 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:14:22.736832 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:14:22.736919 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:14:22.737119 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:14:22.737805 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:14:22.738737 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:14:22.738967 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:14:22.739036 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:14:22.739277 (+ 241us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:14:22.739413 (+ 136us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:14:22.739500 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 01:14:22.739598 (+ 98us) server_negotiation.cc:299] Negotiation successful
0926 01:14:22.739629 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":77,"threads_started":1}
I20260926 01:14:22.744549 23337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:14:22.742150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53666 (local address 127.0.98.61:36663)
0926 01:14:22.742203 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 01:14:22.742205 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:14:22.742265 (+ 60us) server_negotiation.cc:407] Connection header received
0926 01:14:22.742320 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:14:22.742322 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:14:22.742332 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:14:22.742398 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:14:22.742517 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:14:22.743097 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:14:22.743898 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:14:22.744093 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:14:22.744146 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:14:22.744283 (+ 137us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:14:22.744320 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:14:22.744376 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 01:14:22.744431 (+ 55us) server_negotiation.cc:299] Negotiation successful
0926 01:14:22.744460 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 01:14:22.744879 23343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:14:22.741762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38092 (local address 127.0.98.60:37645)
0926 01:14:22.741858 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0926 01:14:22.741860 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:14:22.741918 (+ 58us) server_negotiation.cc:407] Connection header received
0926 01:14:22.741983 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:14:22.741983 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:14:22.741993 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:14:22.742036 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:14:22.742160 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:14:22.742863 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:14:22.743955 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:14:22.744248 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:14:22.744305 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:14:22.744493 (+ 188us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:14:22.744525 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:14:22.744568 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0926 01:14:22.744786 (+ 218us) server_negotiation.cc:299] Negotiation successful
0926 01:14:22.744810 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 01:14:22.745247 23348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:14:22.741731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44596 (local address 127.0.98.62:42353)
0926 01:14:22.741871 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0926 01:14:22.741873 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:14:22.742081 (+ 208us) server_negotiation.cc:407] Connection header received
0926 01:14:22.742220 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:14:22.742221 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:14:22.742292 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:14:22.742394 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:14:22.742619 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:14:22.743232 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:14:22.744227 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:14:22.744470 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:14:22.744522 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:14:22.744877 (+ 355us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:14:22.744984 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:14:22.745042 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 01:14:22.745138 (+ 96us) server_negotiation.cc:299] Negotiation successful
0926 01:14:22.745161 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":59,"threads_started":1}
I20260926 01:14:22.746918 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 22984
I20260926 01:14:22.752069 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 23059
I20260926 01:14:22.756781 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzzHEKz/build/release/bin/kudu with pid 23134
2026-09-26T01:14:22Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2759 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5496 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (269187 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST